robots.txt

The Government Forgot That People Use Google to Navigate the Internet

Photo: Screenshot via TurboTax

Did you know that you can file your taxes online for free? Maybe you did not know this, which is understandable, because the companies that provide these free options would rather you pay them instead. ProPublica reports that the free-filing options from Intuit, the maker of TurboTax, and H&R Block are being intentionally masked from search engines, making it more difficult for people who might need the service to find it.

Most websites have what is known as a robots.txt file, a standard that tells search-engine bots that automatically trawl the web which parts of a site should or should not be indexed. Setting a page to “noindex” in the robots.txt file tells search engines not to include that page in their search results. That same “noindex” command can also be specified directly on a web page itself in a special HTML element.

These privatized Free File options exist thanks to a deal that was struck with the federal government. The IRS agreed not to create its own filing tool, provided that Intuit and others offer free options. According to ProPublica, 70 percent of taxpayers are eligible to use the free tools, but only 3 percent do. Tax-preparation companies have routinely lobbied to keep the U.S. tax code complex and unnavigable, and Congress is currently considering legislation making the Free File arrangement permanent.

“Steering eligible taxpayers away from filing for free or blocking the Free File page from search results violates the spirit of the agreement and calls into serious question the justification for the program,” Ron Wyden, the ranking Democrat on the Senate Finance Committee, told ProPublica.

Of course, the spirit of the law means absolutely nothing if the letter of the law is more lucrative. “Our Free File Alliance offering, like all other alliance partners, is presented in the IRS site and easily reachable through the IRS, on HRBlock.com, and also by googling ‘FFA H&R Block,’” the company told ProPublica. They’re not wrong; those are valid ways to find out where the Free File option is (though the search-query option didn’t work when ProPublica tried it). Point is: If there’s a loophole, someone will find it. These software providers were required to make a free option accessible to the public; the precise avenue that people used to find that option is not specified.

What Intuit and H&R Block are doing here is awful and shady as hell (terms that could also describe most of the freemium tax-prep-software industry, and that’s a bigger issue), but perhaps the primary mistake here was in not considering how people navigate the web. They use search engines. Nobody goes to the IRS’s website to read up on filing options; they search “file taxes free online” or something like that, and click one of the top three results.

This puts Google (and other search engines but we all know it’s mostly Google) in a very important civic position, one that government legislation should address moving forward. It’s a relatively easy fix! Every time the government funds a relevant piece of digital bureaucracy, they should require that it can’t be hidden from search engines. After all, robots.txt is not a proprietary piece of technology, it’s just a standard marker used across the web and interoperable across many systems. Requiring the “robots” not be misused is easy (theoretically; you know how Congress is sometimes).

This solution clearly doesn’t solve the larger problem of government infrastructure being farmed out to profit-driven private industry acting on behalf of its own interests and not the general public’s. Luckily, there’s a fix for that too: Shut down all of the tax-prep companies and throw their CEOs in jail. I’m kidding! Or am I? I am! Maybe.

Tax-Prep Companies Hide Free Options From Search Results