March 29, 2024

Computer Algorithms Rely Increasingly on Human Helpers

But increasingly, behind the curtain there is a decidedly retro helper — a human being.

Although algorithms are growing ever more powerful, fast and precise, the computers themselves are literal-minded, and context and nuance often elude them. Capable as these machines are, they are not always up to deciphering the ambiguity of human language and the mystery of reasoning. Yet these days they are being asked to be more humanlike in what they figure out.

“For all their brilliance, computers can be thick as a brick,” said Tom M. Mitchell, a computer scientist at Carnegie Mellon University.

And so, while programming experts still write the step-by-step instructions of computer code, additional people are needed to make more subtle contributions as the work the computers do has become more involved. People evaluate, edit or correct an algorithm’s work. Or they assemble online databases of knowledge and check and verify them — creating, essentially, a crib sheet the computer can call on for a quick answer. Humans can interpret and tweak information in ways that are understandable to both computers and other humans.

Question-answering technologies like Apple’s Siri and I.B.M.’s Watson rely particularly on the emerging machine-man collaboration. Algorithms alone are not enough.

Twitter uses a far-flung army of contract workers, whom it calls judges, to interpret the meaning and context of search terms that suddenly spike in frequency on the service.

For example, when Mitt Romney talked of cutting government money for public broadcasting in a presidential debate last fall and mentioned Big Bird, messages with that phrase surged. Human judges recognized instantly that “Big Bird,” in that context and at that moment, was mainly a political comment, not a reference to “Sesame Street,” and that politics-related messages should pop up when someone searched for “Big Bird.” People can understand such references more accurately and quickly than software can, and their judgments are fed immediately into Twitter’s search algorithm.

“Humans are core to this system,” two Twitter engineers wrote in a blog post in January.

Even at Google, where algorithms and engineers reign supreme in the company’s business and culture, the human contribution to search results is increasing. Google uses human helpers in two ways. Several months ago, it began presenting summaries of information on the right side of a search page when a user typed in the name of a well-known person or place, like “Barack Obama” or “New York City.” These summaries draw from databases of knowledge like Wikipedia, the C.I.A. World Factbook and Freebase, whose parent company, Metaweb, Google acquired in 2010. These databases are edited by humans.

When Google’s algorithm detects a search term for which this distilled information is available, the search engine is trained to go fetch it rather than merely present links to Web pages.

“There has been a shift in our thinking,” said Scott Huffman, an engineering director in charge of search quality at Google. “A part of our resources are now more human curated.”

Other human helpers, known as evaluators or raters, help Google develop tweaks to its search algorithm, a powerhouse of automation, fielding 100 billion queries a month. “Our engineers evolve the algorithm, and humans help us see if a suggested change is really an improvement,” Mr. Huffman said.

Katherine Young, 23, is a Google rater — a contract worker and a college student in Macon, Ga. She is shown an ambiguous search query like “what does king hold,” presented with two sets of Google search results and asked to rate their relevance, accuracy and quality. The current search result for that imprecise phrase starts with links to Web pages saying that kings typically hold ceremonial scepters, a reasonable inference.

Her judgments, Ms. Young said, are “not completely black and white; some of it is subjective.” She added, “You try to put yourself in the shoes of the person who typed in the query.”

I.B.M.’s Watson, the powerful question-answering computer that defeated “Jeopardy” champions two years ago, is in training these days to help doctors make diagnoses. But it, too, is turning to humans for help.

To prepare for its role in assisting doctors, Watson is being fed medical texts, scientific papers and digital patient records stripped of personal identifying information. Instead of answering questions, however, Watson is asking them of clinicians at the Cleveland Clinic and medical school students. They are giving answers and correcting the computer’s mistakes, using a “Teach Watson” feature.

Article source: http://www.nytimes.com/2013/03/11/technology/computer-algorithms-rely-increasingly-on-human-helpers.html?partner=rss&emc=rss

It Pays to Be on the V.I.P. List When Resolving a Credit Error

The three major agencies, Equifax, Experian and TransUnion, keep a V.I.P. list of sorts, according to consumer lawyers and legal documents, consisting of celebrities, politicians, judges and other influential people. Those on the list — and they may not even realize they are on it — get special help from workers in the United States in fixing mistakes on their credit reports. Any errors are usually corrected immediately, one lawyer said.

For everyone else, disputes are herded into a largely automated system. Their complaints are often electronically ferried to a subcontractor overseas, where a worker spends, on average, about two minutes figuring out the gist of the matter, boiling it down to a one-to-three-digit computer code that signifies the problem — “account not his/hers,” for example — and sending a dispute form to the creditor to investigate. Many times, consumer advocates say, the investigation translates to a perfunctory check of its records.

“The legal responsibility of the credit reporting agencies and of the creditors is well established,” said Leonard Bennett, a consumer lawyer in Newport News, Va. “There is a requirement that they do meaningful research and analysis, and it is almost never done.”

Consumers who have trouble fixing errors through the dispute process can quickly find themselves trapped in a Kafkaesque no man’s land, where the only escape is through the court system.

“You are guilty before you are proven innocent in a situation like this,” said Catherine Taylor, 45, of Benton, Ark., who said she had been denied employment and credit because her filing was mixed up with a felon who had the same name and birthday.

Judy Johnson of Bossier City, La., was confused with a less creditworthy Judith Johnson, with a similar address and Social Security number. For nearly seven years, Judy Johnson, a 63-year-old credit manager for a building supply company, said she tried to remove the black marks from her credit report. But when she was denied a credit card, she knew the problem had returned — a third time. “This time, I was livid,” she said.

She ultimately brought a suit against one of the bureaus, and recently settled for an amount she cannot disclose. But the problems still linger. A deputy sheriff recently came to her door to serve her papers for a debt she says she does not owe.

The credit rating bureaus, private-sector companies that each attempt to track all American consumers’ credit use, have grown much more powerful over the last couple of decades as credit has become a crucial cog in the nation’s financial system. Their reports are used to formulate the all-powerful credit score, which lenders use to determine creditworthiness.

But as the bureaus’ work has become more important, consumer advocates say, regulation has not kept up, in large part because their overseer, the Federal Trade Commission, lacks broad authority. That could change once responsibility for the credit bureaus shifts to the new Consumer Financial Protection Bureau, which will be able to write rules and examine the credit agencies’ policies.

The bureaus, meanwhile, do not have an economic incentive to improve the system, consumer advocates say, because their main customers are the creditors, not consumers.

“There is no neutrality in the credit reporting agencies,” said John Ulzheimer, who has been an expert witness in more than 80 credit-related cases and is president of consumer education at SmartCredit.com. “They work for the lenders who buy credit reports from them, and anyone who suggests otherwise is not being intellectually honest.”

When asked about the V.I.P. category, TransUnion said all consumers “have the ability to speak to a live representative.” Equifax said consumers who received a free copy of their credit report were provided with a number for customer service.

Experian denied that it had V.I.P. lists. But a spokeswoman did say that prominent people deemed high risk — like politicians in an election year — might have their credit files taken offline so that creditors or other companies making inquiries could not get access without the bureau’s permission. Experian said those people did not receive any other special handling.

David Szwak, a consumer lawyer in Shreveport, La., who has handled dozens of credit cases, said that the V.I.P. designation and preferential treatment did exist at Experian, and he provided sworn testimony from former Experian employees that the category existed.

Estimates of credit reports with serious errors vary widely, anywhere from 3 to 25 percent. A recent study, paid for by the Consumer Data Industry Association, the trade group for the bureaus, found potential errors in 19.2 percent of reports, but said that less than 1 percent of them had disputes that, when settled, resulted in a meaningful increase in scores. Even 1 percent translates into millions of consumers, since there are at least 200 million files at each of the bureaus.

The F.T.C. is expected to deliver a nationwide study on credit report accuracy next year that could provide more clarity. It could also include recommendations for legislative action.

The volume of disputes has been rising as consumers borrow more and gain greater access to credit reports. The automated system was a response to that. A spokesman for the trade group said most consumers received an answer within 14 days.

Experian is the only bureau that still processes disputes in the United States, experts said, though most complaints wind their way through the same online system — unless the dispute involves a V.I.P.

“They get a lot more high-end treatment,” said Mr. Szwak, the lawyer, who has read the bureaus’ internal procedure manuals and deposed or cross-examined employees. The biggest difference at TransUnion and Equifax, lawyers said, is that V.I.P.’s disputes are specially handled domestically. Regular consumers’ files, meanwhile, may get priority treatment if they involve a time-sensitive issue, like a mortgage pending, or if the consumer is represented by a lawyer or dealing with fraud.

Last year, new rules went into effect to strengthen existing regulations on the accuracy of reports. The rules also allow consumers to dispute errors directly with the creditor. But critics say the rule lacks any teeth because consumers don’t have the right to sue the companies. (Individuals can, however, sue the bureaus and creditors after lodging a dispute through their system.)

But the problem, advocates say, is that consumers cannot vote with their feet. “They cannot remove their information from the bureaus,” said Chi Chi Wu, a staff lawyer at the National Consumer Law Center, who wrote a report on the automated dispute process in 2009, “or take their business elsewhere.”

Article source: http://feeds.nytimes.com/click.phdo?i=039b2224b1cba11588852eae55e62338