Sales of George Orwell’s Nineteen Eighty-Four have risen since Edward Snowden revealed how the National Security Agency (NSA) of the United States gains access to telephone records and data from technology companies.

So far, if people do not exactly love Big Brother, they are prepared to accept some invasion of their privacy in return for security.

What about “big data”? Companies that hold rapidly expanding amounts of personal information are using new kinds of data analysis and artificial intelligence to shape products and services, and to predict what customers will want.

Mr Larry Page, Google’s Chief Executive, describes his ideal form of technology as “a really smart assistant doing things for you so you don’t have to think about it”.

The vision of living in a virtual Downton Abbey, with a computer to plan your day, suggests the best route to travel, the films you might want to watch and the best flight to catch — even to book it for you — has an allure.

We are all pressed for time and want an easy life. Instead of being bombarded with information and forced to choose, it’s nice to get personal service.

But just as the NSA disclosures have taken people by surprise, although the agency has existed for 60 years, I doubt whether many grasp either the size of the data trail they create daily, or the advances in technology that are permitting a select group of big data enterprises to exploit it.

Technology is evolving so quickly that what was unthinkable two years ago is routine.

“It is both a wonderful and scary future. Companies with huge amounts of data will know more about you than yourself. They will be able to predict what you might do next,” says Mr Lee Kai-Fu, a Beijing-based investor and the former head of Google in China.

LEARNING MUCH FROM SCRAPS OF INFORMATION

In an earlier column, I compared Google to General Electric in the late 19th century — an innovative industrial enterprise riding a wave of new technology. On the flip side of that is Google, Amazon, Microsoft and other technology giants are amassing powers that need to be controlled carefully.

The NSA and big data companies put their databases and computing power to different uses — one to identify spies and terrorists, and the other is to match services to users. They have in common the use of very large databases and techniques such as pattern recognition and network analysis.

At the advanced end, this shades into artificial intelligence of the kind that, for example, intuits what you meant to search even when you misspell key words; it can translate speech into another language in real time (as Microsoft demonstrated in China last year); or learns to recognise a photograph of a cat by viewing thousands of images.

The ability of computers to learn in a similar manner to humans is known as “deep learning” and it is notable that Google has hired several pioneers in the field, including scientist and author Ray Kurzweil.

Among the technology transfers offered by the NSA to private US companies are “cutting-edge machine learning technologies”. Such software can infer a lot from scraps of information, provided that it has enough of them, as shown by the NSA’s effort to analyse phone call metadata from Verizon (and perhaps other operators).

US President Barack Obama assured Americans that “no one is listening to your phone calls”, but this alone is a trove.

A study by Harvard University professor Latanya Sweeney, found that 87 per cent of people can be identified simply by knowing their age, gender and postcode, if these are cross-checked against public databases. That is typical of the data collected by social networks and Internet companies.

VAST DATABASE OF INTENTIONS

The extraordinary power of big data companies comes from being able to combine the personal data of customers with observations about them — from which products they buy to where (as measured by global positioning satellite data from mobile phones) they are. That produces a set of “inferred data” about what they probably want.

If I search on an Android phone for the Taj Mahal while standing in India, for example, Google will prioritise results for the shrine in Uttar Pradesh. If I do the same in Brick Lane, east London, it will suggest local Bangladeshi restaurants. How long before it offers to book a restaurant, based on how I rated others, as I walk around a foreign city at dusk?

At one level, I would be pleased if it did (as long as it was a good one) since it would save me doing the work myself. At another, as a World Economic Forum report on personal data put it: “Inferred data can feel like an all-knowing Big Brother watching the security camera.”

One of the concerns that springs from this is that big data companies with such software are very difficult to compete with. The more data that I and other users provide them with, the better they are at predicting what we want. The machine brain becomes cleverer with use.

Another is trust. Social networks have been poor at protecting data of users, and they hold only a fraction of the information on people’s behaviour, habits and intentions on the new generation of services. It is no wonder that the NSA turns to them — it has computing power and they have swaths of material.

A third is ownership. We each have rights over our own information, but what happens when it gets mixed up with that of others and combined into a vast database of intentions? If I change my mind, how can it be unscrambled?

Above all, we do not know what this technology means because we are only at the beginning of the era of big data. There are plenty of aspects to admire, but it will take some time to love.

ABOUT THE AUTHOR

John Gapper is business columnist and associate editor of the Financial Times.