{"id":39397,"date":"2023-05-16T15:27:05","date_gmt":"2023-05-16T19:27:05","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=39397"},"modified":"2023-05-17T15:41:38","modified_gmt":"2023-05-17T19:41:38","slug":"go-compiler-telemetry","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2023\/05\/16\/go-compiler-telemetry\/","title":{"rendered":"Go Compiler Telemetry"},"content":{"rendered":"<p><a href=\"https:\/\/www.theregister.com\/2023\/02\/10\/googles_go_programming_language_telemetry_debate\/\">Thomas Claburn<\/a> (<a href=\"https:\/\/news.ycombinator.com\/item?id=34744948\">Hacker News<\/a>):<\/p>\n<blockquote cite=\"https:\/\/www.theregister.com\/2023\/02\/10\/googles_go_programming_language_telemetry_debate\/\"><p>Russ Cox, a Google software engineer steering the development of the open source Go programming language, has presented a possible plan to implement telemetry in the Go toolchain.<\/p><p>However many in the Go community object because the plan calls for telemetry by default.<\/p><p>[&#8230;]<\/p><p>Cox&rsquo;s <a href=\"https:\/\/github.com\/golang\/go\/discussions\/58409\">proposal<\/a> summarized lengthier documentation in <a href=\"https:\/\/research.swtch.com\/telemetry-intro\">three<\/a> <a href=\"https:\/\/research.swtch.com\/telemetry-design\">blog<\/a> <a href=\"https:\/\/research.swtch.com\/telemetry-uses\">posts<\/a>.<\/p><\/blockquote>\n\n<p><a href=\"https:\/\/research.swtch.com\/telemetry-intro\">Russ Cox<\/a>:<\/p>\n<blockquote cite=\"https:\/\/research.swtch.com\/telemetry-intro\">\n<p>This post is about why I believe telemetry is important for open-source projects, and what it might look like to approach telemetry in an open-source-friendly way. That leads to a new design I call <em>transparent telemetry<\/em>.<\/p>\n<p>[&#8230;]<\/p>\n<p>Statistics gathered by transparent telemetry make it possible for developers to notice that something is going wrong even when users do not.<\/p>\n<p>\nFor example, during the Go 1.14 release process in early 2020 we made a change\nto the way macOS Go distributions are built, as part of keeping them acceptable\nto Apple&rsquo;s signing tools.\nUnfortunately, the way we made the change also made all\nthe pre-compiled <code>.a<\/code> files shipped in the distribution appear\nstale to builds. [&#8230;] No one reported the bug over three years of Go releases. We didn&rsquo;t notice and fix the problem until late 2022 while investigating something else.<\/p>\n<p>[&#8230;]<\/p>\n<p>\nSome open-source projects have adopted traditional telemetry,\nwith mixed success and varying levels of user pushback.\nFor example: <a href=\"https:\/\/www.theregister.com\/2021\/05\/07\/audacity_telemetry\/\">Audacity<\/a>,\n<a href=\"https:\/\/www.zdnet.com\/article\/gitlab-backs-down-on-planned-telemetry-changes-forced-tracking\/\">GitLab<\/a>,\nand\n<a href=\"https:\/\/news.ycombinator.com\/item?id=11566720\">Homebrew<\/a>.\nHomebrew&rsquo;s telemetry seems to be generally accepted by users,\nand VS Code&rsquo;s detailed telemetry has not stopped\nit from being used by 74% of developers,\nas reported by the <a href=\"https:\/\/survey.stackoverflow.co\/2022\/#integrated-development-environment\">2022 StackOverflow survey<\/a>.<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/research.swtch.com\/telemetry-design\">Russ Cox<\/a> (<a href=\"https:\/\/news.ycombinator.com\/item?id=34750782\">Hacker News<\/a>):<\/p>\n<blockquote cite=\"https:\/\/research.swtch.com\/telemetry-design\">\n<ul>\n<li>\nCounting: Go toolchain programs store counter values in per-week files maintained locally.\n<\/li><li>\nConfiguration: There is a reviewed public process for defining a new graph or metric to track\nand publish on the Go web site. The exact counters that need to be collected,\nalong with the sampling rate needed for high accuracy results, are derived from\nthis configuration.\n<\/li><li>\nReporting: Once a week, an automated reporting program\nrandomly decides whether to fetch the current configuration and\nthen whether to be one of the sampled systems that week.\nIf so, it reports the counters listed in the configuration to a server run by the Go team at Google.\nIn typical usage, we expect a particular Go installation to report each week with under 2% probability,\nmeaning less than once per year on average.\n<\/li><li>\nPublishing: The server publishes each day&rsquo;s reports in full (in a compressed form)\nas well as publishing the tabular and graphical summaries defined in the configuration.\n<\/li><\/ul>\n<\/blockquote>\n\n<p><a href=\"https:\/\/research.swtch.com\/telemetry-uses\">Russ Cox<\/a>:<\/p>\n<blockquote cite=\"https:\/\/research.swtch.com\/telemetry-uses\">\n<p>Understanding how often GOPATH mode (non-module mode) is used is important for understanding how important it is to keep running, and for which use cases.<\/p>\n<p>[&#8230;]<\/p>\n<p><\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/circumstances.run\/@davidgerard\/109848768306100450\">David Gerard<\/a> (via <a href=\"https:\/\/news.ycombinator.com\/item?id=34771472\">Hacker News<\/a>):<\/p>\n<blockquote cite=\"https:\/\/circumstances.run\/@davidgerard\/109848768306100450\"><p>the Go maintainers are now claiming that objectors to Google&rsquo;s opt-out telemetry proposal on the Go compiler - yes, really - are arguing in bad faith and violating the Code of Conduct, and their comments are getting hidden.<\/p><\/blockquote>\n\n<p><a href=\"https:\/\/github.com\/golang\/go\/discussions\/58409\">Russ Cox<\/a>:<\/p>\n<blockquote cite=\"https:\/\/github.com\/golang\/go\/discussions\/58409\"><p>The current plan is to use <a href=\"https:\/\/research.swtch.com\/telemetry-opt-in\">some form of opt-in<\/a> to collect telemetry.<\/p>\n<p>[&#8230;]<\/p>\n<p>\nAs mentioned in the <a href=\"telemetry-intro\">introductory post<\/a>,\n<code>-buildmode=shared<\/code> has never worked particularly well,\nand its current design is essentially incompatible with Go modules.\nWe have <a href=\"https:\/\/github.com\/golang\/go\/issues\/47788#issuecomment-954890659\">a rough design<\/a>\nfor how we might make it work, but that&rsquo;s a lot of work, and it&rsquo;s unclear\nwhether it should be a high priority.\nOur guess is that there are essentially no users of this flag,\nand that therefore it shouldn&rsquo;t be prioritized.\nBut that&rsquo;s just a guess; data would be better.<\/p>\n<p>[&#8230;]<\/p>\n<p>The Go build cache is a critical part of the user experience,\nbut we don&rsquo;t know how well it works in practice.\nThe original build cache heuristics were decided in 2017 based on\na tracing facility that we <a href=\"https:\/\/go.dev\/issue\/22990\">asked users to run for a few weeks<\/a>\nand submit traces.\nSince then, many things about Go builds have changed,\nand we have no idea how well things are working.<\/p><\/blockquote>\n\n<p>See also: <a href=\"https:\/\/www.dropboxforum.com\/t5\/Integrations\/Why-So-Much-Telemetry\/td-p\/455961\/page\/3\">Dropbox<\/a> (via <a href=\"https:\/\/news.ycombinator.com\/item?id=35724939\">Hacker News<\/a>).<\/p>\n\n<p>Previously:<\/p>\n<ul>\n<li><a href=\"https:\/\/mjtsai.com\/blog\/2023\/04\/26\/1password-to-add-telemetry\/\">1Password to Add Telemetry<\/a><\/li>\n<\/ul>\n\n<p id=\"go-compiler-telemetry-update-2023-05-17\">Update (2023-05-17): <a href=\"https:\/\/github.com\/golang\/go\/issues\/58894#issuecomment-1505888051\">Russ Cox<\/a> (via <a href=\"https:\/\/news.ycombinator.com\/item?id=35974653\">Hacker News<\/a>):<\/p>\n<blockquote cite=\"https:\/\/github.com\/golang\/go\/issues\/58894#issuecomment-1505888051\"><p>No change in consensus, so <strong><a href=\"https:\/\/go.dev\/s\/proposal-status#accepted\">accepted<\/a><\/strong>. &#x1F389;<\/p><\/blockquote>","protected":false},"excerpt":{"rendered":"<p>Thomas Claburn (Hacker News): Russ Cox, a Google software engineer steering the development of the open source Go programming language, has presented a possible plan to implement telemetry in the Go toolchain.However many in the Go community object because the plan calls for telemetry by default.[&#8230;]Cox&rsquo;s proposal summarized lengthier documentation in three blog posts. Russ [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"apple_news_api_created_at":"2023-05-16T19:27:12Z","apple_news_api_id":"80df636b-afa5-48f0-af6d-9b76953e0d6b","apple_news_api_modified_at":"2023-05-17T19:41:41Z","apple_news_api_revision":"AAAAAAAAAAAAAAAAAAAAAg==","apple_news_api_share_url":"https:\/\/apple.news\/AgN9ja6-lSPCvbZt2lT4Naw","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_is_hidden":false,"apple_news_is_paid":false,"apple_news_is_preview":false,"apple_news_is_sponsored":false,"apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":"\"\"","apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"footnotes":""},"categories":[4],"tags":[466,255,47,846,51,991,355,71],"class_list":["post-39397","post","type-post","status-publish","format-standard","hentry","category-programming-category","tag-codesigning","tag-compiler","tag-dropbox","tag-go-language","tag-google","tag-open-source-software","tag-privacy","tag-programming"],"apple_news_notices":[],"_links":{"self":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/39397","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/comments?post=39397"}],"version-history":[{"count":4,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/39397\/revisions"}],"predecessor-version":[{"id":39412,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/39397\/revisions\/39412"}],"wp:attachment":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/media?parent=39397"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/categories?post=39397"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/tags?post=39397"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}