AnandWrites: This is Bloomberg saying it. Plute-on-plute violence, and true. "Facebook is a red state."

If you're a Democrat in elected office and you haven't yet been as tough on Facebook's existential threat to democracy, regardless of whether a child of yours worked there, wake up and don't be outflanked by Bloomberg.

I mean, Mike Bloomberg is a threat to democracy also. So when he calls someone else a threat to democracy, they're really, really a threat to democracy. It's like squaring a number.

Your occasional reminder that the path to a freer and more just America runs through more plute-on-plute beefing and less beefing among the rest of us.

Previously, previously, previously, previously, previously.

A few years back I was excited to learn that there was a video that would teach me how to remove the seeds from a pomegranate in ten seconds.

I was less excited when I learned that the video was four and a half minutes long. It shouldn't take more than sixty seconds to explain how to do something that only takes ten seconds to demonstrate. Maybe something like this:

“Here's how to seed a pomegranate in ten seconds: Cut the pomegranate like this.”

THUNK THUNK

“Hold half of the pomegranate over a bowl. Take a wooden spoon.”

WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP

“Now the other half.”

WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP WHAP

“There you go! Thanks for watching.”

Instead, this guy spent two minutes up front building suspense and telling us how awesome this was going to be once he finally got around to showing it to us. Then he spent another minute on the back end waffling around instead of just turning off the camera.

Many years ago I gave a conference talk in which I complained that conference speakers waste everyone's time introducing the subject before they work around to the point. I wish I'd used that pomegranate video as an example.

God I hate tech hyperbole. And I literally hate the use of “literally” to mean its opposite … but let me explain how bad technology product management can literally lead to the end of the world.

Donald Trump represents an existential threat to humanity. To put such a man at the helm of the world’s most powerful nation is like handing over the controls of a nuclear submarine to a petulant baby. That’s a poor simile only because it’s not an analogy but a nearly literal description.

What can I do about this? I’m just one person, one vote. Moreover, I’m in California, which will surely vote Clinton anyway, so my vote won’t sway the outcome. I could advocate, I could preach to everyone around me, but really most of the people in physical proximity to me already agree with me.

What about technology? I’m in the center of Silicon Valley, I know me some techmology, can’t I do something wizardly to extend the power of a single voice? Nope. I mean, I can write this little essay, and maybe my fifty readers will like it, but those fifty people and everyone they’ll share it with already agree with me.

But about twenty miles from here, there are a couple of dozen people who literally hold the fate of our political conversation in their hands. In fact, it’s been in their hands for quite some time now, and they’ve made decisions which, only in retrospect, appear to have been disastrous for our nation’s politics.

At Facebook, the News Feed is the main stream of information that people see when they use the service. It has become the single most important source of news and conversation for many if not most Americans. It is designed to show people information that they want, which largely means showing people what they already agree with, from people who they already are inclined to sympathize with.

At Google, the search results page answers billions of queries each day, from billions of people. The results are carefully shaped not just with regard to each query, but as much as possible conformed for the particular user, so that the user sees results they are more likely to want to click, which in essence means showing them information they already agree with.

I’m not the first to note that the creation of these echo chambers only serves to reinforce existing biases, and isolate people from diverse opinions that could broaden their horizons and enrich our society. I might be among the first to charge that the product managers who now lead Facebook News Feed and Google Search are failing at their jobs.

On its face that’s a ridiculous statement, as we are talking about two of the most successful products in history, literal world changers. And who could argue against the general strategy of conforming experience to user tastes? But there comes a time in the life cycle of even massively successful products, when the product has attained a use and effect that were never anticipated through all of the prior success. Product managers who do not grapple with what their products have become, in all dimensions, are not doing their jobs well.

News Feed and Search are unique in landscape of all products. These are no longer simply things that people use, and therefore need to be designed to be as pleasant and popular as possible. These products now form the infrastructure of political conversation, they have become the backbone of our polity, they are the means by which citizens of our nation engage with each other on the essential ideas of community. The success of these products must now be judged on how well they serve beneficial outcomes in our society, especially our politics.

There are plenty of people at Facebook and Google who are deeply invested in denying this responsibility, which is so self-evident to all of the rest of us mere users. They would like to say that their products are designed to be “neutral,” to simply follow algorithms that have no sense of society or humanity. They want to hide their power behind obfuscating explanations of math and probability.

Some of this may be a difference in perspective. Some of this may be benign short-sightedness. But some of it is moral cowardice. I hate to make such an inflammatory charge, but when you have the ability to shape a product in a way that would reduce the likelihood of a fascist from taking the reins of a country with the firepower to end life as we know it, and you deny that you have this power, I have a hard time calling this anything other than what it is.

Facebook and Google know that their products contribute to a stifled political conversation that only hardens lines of hate and allows well-meaning people to isolate themselves in their own safe spaces. Will they continue to build their products in a way that divides our society? Or will they take real moral responsibility for how their products shape our political conversation, and make their products a conduit for uncomfortable ideas that could improve our world? Will they break down the barriers between hardened positions, expose ignorance to truth, measure hatred and inject love? Or will they claim that these goals are too soft, and anyway achieving them is too hard?

When Philip Morris discovered that their product was killing their customers, they hid the evidence for as long as they could, and they denied the truth even after it was apparent to everyone else, all so they could squeeze out the last dollars from their death-dealing empire. When Coca-Cola realized that sugary drinks were contributing to unprecedented rates of obesity, they diversified their product lines to include healthy drinks as well as sugar bombs – not exactly admirable, but at least preparing for a shift where people who could watch out for themselves would continue to contribute to the company’s bottom line. At this point, I would be okay with lesser evils, but I would prefer to see moral courage.

Photo by Tiago Aguiar, public domain/CC0.

The analytics team at the Wikimedia Foundation is excited to release a new dataset for our community and the world: unique devices. This is a new way for us to estimate how many people read Wikimedia projects on the web on a monthly and daily basis. Our measure is an estimate because we are counting devices, rather than visitors, and some people use more than one device to access the web (a mobile phone and a desktop, for example) and some people share computers. While unique devices does not equal unique visitors, it is a good proxy for that metric, meaning that a major increase in the number of unique devices comes likely from an increase in visitors.

In developing this dataset, we looked critically at previous methods we’ve used to measure reach. We knew we wanted to count unique readers and we wanted to do it in a manner that treated user privacy and security as a priority. In response, we came up with a novel, light-touch way of estimating unique device counts for Wikimedia projects. In this post, we’ll share how we used to estimate unique readers, why we needed to change, and how we’re doing it now.

We invite you to explore this new dataset and hope it’s helpful for the Wikimedia community in better understanding our projects. This data can help measure how large is the reach of Wikimedia projects on the web. For example, we have estimated more than 600 million unique devices in January 2016 on English wikipedia alone (adding desktop and mobile). We expect this number to vary seasonally greatly as numbers for November were around 440 million. Still, English Wikipedia is the largest project, with three times as many devices as the next most popular project, Spanish Wikipedia.

Photo by Nuria Ruiz, CC BY-SA 3.0.

How we used to count unique web readers

Users of Wikimedia projects don’t need to login to access content, so unlike other major websites, we can’t count unique readers via accounts. Since 2009, the Wikimedia Foundation used comScore to report data about unique visitors. The relationship was pro bono, and we benefited from having an estimate of our traffic and ranking among other websites. In January 2016, however, we decided to stop reporting comScore numbers because of certain limitations in the methodology.

ComScore’s strength is in measuring traffic from desktop, and we found that this data was less accurate for mobile devices, which represent a significant portion of Wikimedia traffic. This could be addressed by installing tracking beacons on our sites that stream data back to comScore, but we didn’t do this because we aim to protect user privacy and security, and, we do not share user data with third parties not subjected to the same standards.

With the unique devices dataset, we’ve been able to quantify the shift to mobile across all projects. In almost all Wikimedia projects, more than half of our unique devices are accessing content using the mobile sites. That means that in terms of usage, mobile represents more than half of Wikimedia projects’ overall usage, which makes an accurate measurement for mobile traffic a high priority. In some projects, the mobile access represents a much higher percentage. For example, 85% of usage of Indonesian Wikipedia is happening in the mobile site rather than desktop .

How we count unique readers now. Privacy by Design.

Developing our own way to count allows us to be certain about the meaning behind the numbers, to understand how the approach holds up in practice, and to share the high-level numbers with the wider community. Our goal when we started was to acquire greater value from usage data while reducing the risk of data misuse (i.e. privacy risks).

Imagine we want to count the number of patients a doctor has in a month without disclosing sensitive data about medical visits of specific patients. An intrusive way to do this would be keeping a list of users that came to the doctor’s office that month. For example:

1. Alicia
2. Susana
3. Maria

A list like that holds a lot of personal information. Alicia might not be very fond of everybody knowing that she went to the doctor on March 2016. Another way to to count patients while preserving their anonymity would be making a list like:

1. First Patient
2. Second Patient
3. Third Patient

However, in this case there is no way to differentiate between the First and Third patient once they have left the doctor’s office. The problem is that we want to count patients that come in a given time period (for example, a month), but we do not want to identify patients individually. At the same time, we need to be able to know who has already come to the office that month so we don’t count them twice.

What we do: when patients leave the doctor’s office, we hand them slips that print the date in which they came into the office last. In our example, the slip says: “Last-Time-Visited: February 17th”. How is this useful? Let’s say we are counting unique patients of the doctor office in March: anyone that comes into the doctor’s office without a time slip is a new patient for the month (they have not been at the doctor’s office before at all, we are giving slips to everyone). Anyone that comes with a slip that says “Last-Time-Visited: February 17th” when we are in March should also be counted as a unique patient for the month of March as they were at the doctor’s office last on February. For our counting to be accurate we need to update the time information to note that we have seen this patient already in March, so before patient leaves we update their slip and give them a new one that says “Last-Time-Visited: March 15th”.

The careful reader would have noticed that we do not need to print the day on the timeslip—to count monthly users is sufficient with just printing the month. Now, printing the date more precisely allow us to count daily uniques with the same methodology. If we are counting, say,  the number of patients on March 15th anyone coming without a time slip needs to be counted towards the daily toll (patient has not been at the doctor’s office before), anyone with a time slip with a date before March 15th should be counted too. If a patient comes twice to the office on March 15th, we will not count them twice as we only update their time slip once a day, the second time the patient shows up the slip already says “Last-Time-Visited: March 15th”.

Using this strategy, we have successfully implemented regular unique devices counting in a manner that treats user privacy and security as a priority. We believe that by designing with privacy and security in mind, we can develop products and services that benefit our users, the community and staff alike.

Limitations

As with any method, unique devices has some limitations. It only counts the number of unique devices used to access Wikipedia. This leaves room for duplication:  when a user looks at a Wikipedia page on their desktop and mobile phone, we will be counting two devices.

Bots
Things got a little complicated when we tried to implement this original idea. We set a cookie in every browser request that comes into a Wikimedia project with the value of our time slip: “Last-Access: February 17th”. The cookie is set to expire every 30 days and its value is refreshed in every request. When we started counting we realized that the number of requests with no cookie seemed too high. In hindsight, this makes sense. Wikipedia gets a lot of automated bot traffic that does not accept cookies. In our analogy above, the bots are the people that come into the doctor’s office with no time slip. With our methodology, we cannot tell doctor’s patients from other people that are just walking by into the office to, say, deliver a package. Solving this last problem was not trivial given that our bot detection framework is somewhat ”rustic”. The technical solutions we used are documented at length here and here.

Smaller Projects
We looked at data regarding unique devices for all Wikimedia projects and it was fairly obvious that for small projects (those with fewer than 1000 daily uniques), data for daily uniques was too sparse to be meaningful. Therefore, our public datasets only contain data for projects with more than 1000 uniques.

Drawbacks
While our approach to counting uniques reduces the risk of data misuse, it has some drawbacks from an analytic perspective. For example, we cannot use the methodology to split up our user-base for an A/B test. We are also (purposely) not counting users who browse Wikimedia projects with their cookies off. And lastly, we know our methodology somewhat under-reports numbers, but we are okay with that.

I want to use this same methodology to count uniques on my site. Can I?

You sure can. The code that sets cookies is deployed to our varnish caching servers rather than the MediaWiki code and it is somewhat dry. You can find it on github. Now, implementing the same logic in php/python/java/you-name-it should be easy.

Nuria Ruiz, Software Engineer
Madhumitha Viswanathan, Software Engineer
Aaron Halfaker, Senior Research Scientist
Wikimedia Foundation

Imagine if your sewer pipe started demanding that you make major changes in your diet.

Now imagine that it got a lawyer and started asking you to sign things.

You would feel surprised.

This is the position I find myself in today with IFTTT, a form of Internet plumbing that has been connecting peacably to my backend for the past five years, but which has recently started sending scary emails.

If you've never heard of it, If-This-Then-That is a service that lets you connect websites together, so that things that happen in one place automatically trigger some regrettable action someplace else. For example, you might write an IFTTT ‘recipe’ that tweets anything you post on Facebook, because you are a monster.

A lot of Pinboard people use IFTTT. Yesterday, they received the following form letter:

Dear username,

We're working on a new IFTTT platform for developers that makes building Channels and Recipes a breeze.

Recently, we've worked with our partners to migrate to the improved platform, but some have chosen not to do so. Unfortunately, the Pinboard Channel did not migrate to the new platform and will be removed on April 4th.

Pinboard is one of our favorite services and we're all sad to see it go. We hope down the road it may be back.

Stay tuned to the latest Channels launching on IFTTT!

— The IFTTT Team

Because many of you rely on IFTTT, and because this email makes it sound like I'm the asshole, I feel I should explain myself.

In a nutshell:

1. IFTTT wants me to do their job for them for free

2. They have really squirrely terms of service

1. Working for Free

A service like IFTTT writes "shim code" that makes it possible to connect online services together like Legos. Everything slots into everything else. This is thankless, detailed work (like developing TurboTax or Dropbox) that when done right, creates a lot of value.

IFTTT has already written all this shim code. They did it when they were small and had no money, so it's difficult to believe they have to throw it away now that they have lots of staff and thirty million dollars.

Instead, sites that want to work with IFTTT will have to implement a private API that can change without warning.

This is a perfectly reasonable business decision. It is always smart to make other people do all the work.

However, cutting out sites that you have supported for years because they refuse to work for free is not very friendly to your oldest and most loyal users. And claiming that it's the other party's fault that you're discontinuing service is a bit of a dick move.

I am all for glue services, big and small. But it's better for the web that they connect to stable, documented, public APIs, rather than custom private ones.

And if you do want me to write a custom API for you, pay me lots of money.

2. Squirrely Terms of Service

The developer terms of service don't seem to be available by a public URL, so I will quote the bits that stung me. I invite IFTTT lawyers to send me a takedown notice, because that will be the funniest part of this fracas so far.

To begin with, IFTTT wants me to promise never to compete with them:

2.You shall not (and shall not authorize or encourage any third party to), directly or indirectly: [...] (xii) "use the Developer Tool or Service in conjunction with a product or service that competes with products or services offered by IFTTT. You hereby make all assignments necessary to accomplish the foregoing.”

Pinboard is in some ways already a direct competitor to IFTTT. The site offers built-in Twitter integration, analogous to IFTTT’s twitter->Pinboard recipe. I don’t know what rights I would be assigning here, but this is not the way I want to find out.

Next, they make a weird claim about owning not just their API and service, but the content that flows through it:

3. Ownership. IFTTT shall own all right, title, and interest (and all related moral rights and intellectual property rights) in and to the Developer Tool, Service, and Content.

They require that I do custom development work for them, for free, on demand:

11. Compatibility. Each Licensee Channel must maintain 100% compatibility with the Developer Tool and the Service including changes provided to you by IFTTT, which shall be implemented in each Channel promptly thereafter.

And they assert the right to patent any clever ideas I have while doing that free work for them, even though I hate software patents:

12. Patent License. Licensee hereby grants IFTTT a nonexclusive, sublicensable, perpetual, fully-paid, worldwide license to fully exercise and exploit all patent rights with respect to improvements or extensions created by or for Licensee to the API

Finally, they reserve the right to transfer this agreement to anyone at all, without my consent:

17.This Agreement is personal to Licensee and may not be assigned or transferred for any reason [...]. IFTTT expressly reserves the right to assign this Agreement and to delegate any of its obligations hereunder.

I say nuts to all that.

I'm sorry your IFTTT/Pinboard recipes are going to stop working.

It's entirely IFTTT's decision to drop support for Pinboard (along with a bunch of other sites). They are the ones who are going to flip the switch on working code on April 4, and they could just as easily flip the switch back on (or even write an IFTTT recipe that does it for them). Weigh their claims about Pinboard being a beloved service accordingly.

For users left stranded, I recommend taking a look at Zapier or Botize, which offer a similar service, or at one of the dozens of new sites that will spring up next week to capture the market that IFTTT is foolishly abandoning.

The “mob” was actually a group of people on bikes trying to flag down a motorist who had hit one of their friends and driven off, dragging his bike. Image: KGTV San Diego

Here’s a great example of how American law enforcement tends to produce perverse results when it comes to traffic collisions.

A cyclist in San Diego was hit by a driver and managed to avoid more serious injury by jumping off his bike. Prior to the incident, the motorist had been honking repeatedly at the group the victim was riding with, according to this report from KGTV San Diego.

Although she struck a person, dragged his bike for blocks, and only stopped when confronted by the victims’ friends, the driver will receive no ticket and face no criminal charges. In fact, one of the friends who chased the driver down may be charged with a misdemeanor for banging on her window and breaking it. In KGTV’s telling, that makes the cyclists a “mob” and the whole incident “a tussle” between them and the driver.

That’s how it goes on American streets: Harming a person with your car carries no penalty but harming someone’s car most definitely does.

Hat tip Shane Phillips.