Archive for November 10, 2017

Friday, November 10, 2017

Building a Better Date/Time Library for Swift

Dave DeLong (tweet):

Chronology is an attempt to build a better date and time API for Swift on top of the constructs provided by Foundation.

Foundation.framework has one of the absolute best and most capable APIs available to any developer on any platform. However, it comes with a heavy price in the form of cognitive load. Too many developers do too many incorrect things. Over the years, Foundation has gained more and more convenience methods via the Calendar type, but fundamental issues remain.

We’re Just Doing Data Entry for Google

André Staltz:

What has changed over the last 4 years is market share of traffic on the Web. It looks like nothing has changed, but GOOG and FB now have direct influence over 70%+ of internet traffic. Mobile internet traffic is now the majority of traffic worldwide and in Latin America alone, GOOG and FB services have had 60% of mobile traffic in 2015, growing to 70% by the end of 2016. The remaining 30% of traffic is shared among all other mobile apps and websites. Mobile devices are primarily used for accessing GOOG and FB networks.


Prior to 2014, Search Engine Optimization (SEO) was a common practice among Web Developers to improve their site for Google searches, since it accounted for approximately 35% of traffic, while more than 50% of traffic came from various other places on the Web. SEO was important, while Facebook presence was nice-to-have. Over the next 3 years, traffic from Facebook grew to be approximately 45%, surpassing the status that Search traffic had. In 2017, the Media depends on both Google and Facebook for page views, since it’s the majority of their traffic.


There is a tendency at GOOG-FB-AMZN to bypass the Web which is motivated by user experience and efficient communication, not by an agenda to avoid browsers. In the knowledge internet and the commerce internet, being efficient to provide what users want is the goal. In the social internet, the goal is to provide an efficient channel for communication between people. This explains FB’s 10-year strategy with Augmented Reality (AR) and Virtual Reality (VR) as the next medium for social interactions through the internet. This strategy would also bypass the Web, proving how more natural social AR would be than social real-time texting in browsers. Already today, most people on the internet communicate with other people via a mobile app, not via a browser.

Via Matt Birchler:

And now we have Google Assistant, which is a great tool for getting information, but is another step in obscuring the line of what content belongs to who. You can ask the Assistant a question and it will give you an answer in the Assistant app (or just in the air if you’re using a Google Home). A recipe, for example, will be scraped from someone’s cooking blog and then presented in the Assistant app as if Google had created this recipe. You can poke around the interface to find out where it came from, and you can sometimes tap a link to see the source of an answer, but it’s not the default behavior. Hell, a “failed state” in Assistant is when it has to show you a list of websites in your search results.

In short, Google once was a tool for getting people to content that we as creators made, but today it seems like we are just doing data entry in Google’s database to let them display nuggets of our content in their software.

Work on SQLite 4 Has Concluded

Richard Hipp (via Hacker News):

Lessons learned from SQLite4 have been folded into SQLite3 which continues to be actively maintained and developed. This repository exists as an historical record. There are no plans at this time to resume development of SQLite4.

The Design Of SQLite4:

SQLite4 stores all content, from all tables and all indices, in a single keyspace. This contrasts with SQLite3 that required a separate keyspace for each table and each index. The SQLite4 storage also differs from SQLite3 in that it requires the storage engine to sort keys is lexicographical order, whereas SQLite3 uses a very complex comparison function to determine the record storage order.


The default built-in storage engine is a log-structured merge database.


SQLite3 allows one to declare any column or columns of a table to be the primary key. But internally, SQLite3 simply treats that PRIMARY KEY as a UNIQUE constraint. The actual key used for storage in SQLite is the rowid associated with each row.

SQLite4, on the other hand, actually uses the declared PRIMARY KEY of a table (or, more precisely, an encoding of the PRIMARY KEY value) as the key into the storage engine. SQLite4 tables do not normally have a rowid (unless the table has no PRIMARY KEY in which case a rowid is created to be the implicit primary key.) That means that content is stored on disk in PRIMARY KEY order. It also means that records can be located in the main table using just a single search on the PRIMARY KEY fields.

LSM Design Overview:

The LSM embedded database software stores data in three distinct data structures[…]


I’ve had the chance to hear Richard Hipp talk about SQLite yesterday! He mentioned that the LSM tree storage engine is available as an extension to sqlite3. More specifically, he mentioned that he didn’t really get the performance improvements he had hoped for, for insertion-heavy use cases.

I think part of this is because of a fundamental limitation of sqlite that it’s an embedded database that has to persist data on disk at all times: The design of LSM trees works well with databases with a resident in-memory component because it’s an approximation of just dumping every new thing you see at the end of an unordered in-memory array. This is as opposed to a data structure like a b-tree where you have to find exactly where to put the data first, and then put it there. This finding bit means you’re doing a lot of random access in memory, which is thrashing all of your caches (CPU / disk etc). LSM trees avoid this thrashing by just dumping stuff at the end of an array. However this means you have to scan that array to do lookups (as opposed to something easier like binary search). Then as your array gets big, you merge and flush it down to a lower “layer” of the lsm tree which is slightly bigger and sorted. And when that one fills, you flush further. And these merge-flushes are nice big sequential writes so that’s nice too.

Anyway, with SQLite, the highest layer of your LSM tree would probably (this is conjecture) have to be on disk because of the way that there is no server component, versus in an in-memory system it’d probably be in your L2/L3 cache or at least your main memory. So this could be one reason why that model didn’t work out as well for them.

Previously: SQLite 4.

See also: Richard Hipp on the Changelog podcast.

How Facebook Figures Out Everyone You’ve Ever Met

Kashmir Hill:

Behind the Facebook profile you’ve built for yourself is another one, a shadow profile, built from the inboxes and smartphones of other Facebook users. Contact information you’ve never given the network gets associated with your account, making it easier for Facebook to more completely map your social connections.

Shadow contact information has been a known feature of Facebook for a few years now. But most users remain unaware of its reach and power. Because shadow-profile connections happen inside Facebook’s algorithmic black box, people can’t see how deep the data-mining of their lives truly is, until an uncanny recommendation pops up.

Facebook isn’t scanning the work email of the attorney above. But it likely has her work email address on file, even if she never gave it to Facebook herself. If anyone who has the lawyer’s address in their contacts has chosen to share it with Facebook, the company can link her to anyone else who has it, such as the defense counsel in one of her cases.


Handing over address books is one of the first steps Facebook asks people to take when they initially sign up, so that they can “Find Friends.”

Update (2017-11-15): Christopher P. Atlan:

God damn Facebook. You need to press learn more to be able to skip the import of your contacts.

AstroPad’s Camera Button Rejected From the App Store

Savannah Reising (via Tim Hardwick):

We planned to introduce the Camera Button in an update to Astropad Studio going out today. However, we are disappointed to report that the Camera Button was rejected by Apple’s App Store review under Section 2.5.9:

Apps that alter or disable the functions of standard switches, such as the Volume Up/Down and Ring/Silent switches, or other native user interface elements or behaviors will be rejected.

It doesn’t look like the guideline actually covers what their app does. But it’s the kind of feature that I would expect Apple to disapprove of.

David Barnard:

I get why Apple rejected the super fun/innovative @astropadapp camera button, but I’m still very disappointed.

And as I’ve said many times before, this kind of rejection has a chilling effect on iOS innovation.

App Review and many other App Store policies end up being shaped primarily by scammers, not conscientious developers.

At this point in the life of the platform I’d much rather Apple make special exceptions than outright block innovation/experimentation.

Apple pretends the App Store is a level playing field, but it never has been. And it’s better for customers if it’s not.

See Uber’s special entitlement for Apple Watch. They couldn’t have built a great app without special treatment.

There’s no way for us to ever know what all has been rejected and what was never even submitted to the App Store for fear of rejection.

Rejecting an innovative app/feature here or there might seem innocuous, but it’s easy to underestimate the long-term consequences.

I would’ve never built the Camera Button because I assumed it would be rejected. And that’s a very real problem.

Previously: The Camera Button.