All instructions to LLMs are merely suggestions to nudge it in the right behaviour. Unless you have a deterministic guardrail that guards against a single specific action, everything else is a slot machine that's biased strongly in your favour.
My global CLAUDE.md explicitly states "When commenting on code and configs, or writing MD files, strictly write within the domain of the content being commented on. DO NOT include information, negatives or ramblings from work sessions. For e.g. if commenting on a proto string field that is replacing an int field, do not comment that 'this is not an int field'".
This reduced the idiocy of the agent (Opus 5 included) when writing documents. But I'm still catching it writing README.md talking about the negatives that it removed. Those belong in the memory if it is actually that important (most of the time it's junk), but Claude doesn't seem to understand and never ever learns.
> But there is one huge question: did Tristan opt out of model training for his ChatGPT and Codex sessions? If the answer is no, then this seems fair game.
Just because something is legal and permitted by terms of service doesn't mean it's morally right.
>Just because something is legal and permitted by terms of service doesn't mean it's morally right.
What are you expecting OpenAI to do exactly if these mathematicians voluntarily submitted their prompts into ChatGPT's training data? Are they supposed to manually review all their data to make sure competing mathematicians didn't accidentally leave the "submit prompts" toggle on?
Or were they supposed to not try to solve Navier-Stokes, or were they supposed to just not tell anyone that they had solved it?
To me it’s morally ambiguous… if you hand parts of your thinking over to a tool like this (knowing full well the terms of service), of course the tool makers will want to claim some credit, and they do deserve it. But the bigger question to me is the scientific one: did their new model arrive at this result because it had closely-related training data from a human, or did it extrapolate to this line of thought on its own? The answer says a lot about how valid their claims of “AGI” are vs. a very fortuitously cherry-picked example.
It would actually be a really interesting study, if they would ever be willing to be transparent about this, how the result differs with and without his conversations in the training set. How quickly it arrives at the result, whether it takes the same approach, etc.
> What are you expecting OpenAI to do exactly if these mathematicians voluntarily submitted their prompts into ChatGPT's training data?
Personally, I would expect them to have a little class, to KYC, and to manually turn off training for known competitors using their service so as to avoid any unforced goofs like this.
> Are they supposed to manually review all their data t
Yes. They should determine if training data included this teams data. Consider the money they spent, the press release and the purpose of their publication.
Since they failed to answer this question they shouldn't have published.
- Good design requires user interviews, which are less common in open source
- The sorts of people who are most attracted to open source work are programmers, not designers
- Open source software can get popular even with bad design; closed source software with bad design fails and shuts down
Designers have it really hard in free software because good design is about understanding the big picture, knowing what to leave out, have product vision to follow.
That's tough to do in environment where goal is to make everyone happy and the biggest say have programmers who are developing the thing who are usually not interested in design, inexperienced in design or have their personal very specific product vision.
I think you definitely identified the issue well. Changing the design of an existing project, especially if you're not already involved in the project is a daunting task. You have to try and convince a core group of users that the design is going to be better to entice new users. But the problem is that those core group of developers likely don't care that the UI isn't up to par, and likely don't want it to change.
I even remember having this problem personally when trying to improve designs of webpages at my workplace. We had many of our technical users still using the command line for their mail, and even for browsing some webpages. Their whole workflow was stuck in the command line and they've been doing it that way for 20+ years. Trying to change things to improve the experience for others was like pulling teeth. They would retort the classic "Why change it? It works the way it is, no need to change it". Which to a degree I can understand. Some websites out there really do change their UI's for completely silly reasons and sometimes even lose functionality because of it.
At least this is one positive thing coming from AI driven development. People can actually make some pretty decent UI's without having to know as much about good UI design. AI can try to utilize the "best practice" standards and make a UI that often looks much better than most devs could make. There are some downsides of course, such as the fact that almost every new website nowadays looks the same, since Claude seems to have one style it always goes with. But maybe that tradeoff is okay.
I don't agree about the AI at all. AI is terrible at design (compared to programming).
Having relatively nice UI skin doesn't equal good design. They are surprisingly unrelated as i am sure people know highly usable well designed software with bad UI and also know super beautiful UIs that are unusable - badly designed.
AIs are bad at design because they don't understand and can't keep context of the whole together. Just like in programming they are much better at solving single well defined problem. Design is very much opposite of that, it's always about reharmonizing in context with the whole after you make a change.
They have the SEO juice to push out and replace dedicated wikis as the top result, even when there are long-standing dedicated wikis for the game/universe. Fandom wikis then typically have maybe half of the information from the dedicated wiki PLUS a bunch of intrusive ads. There's no instance that I know of where a Fandom wiki popping up for a community was a positive unless they simply didn't have one to start with.
Well, I'm an American, but US media is disproportionately set in Los Angeles or NYC, and I haven't been to either of those places.
I was watching Monk recently, and it feels kinda neat seeing a show shot in San Francisco because I used to live there. I wouldn't say it feels "cheap" though.
Pangram is a complete scam. There is no durable way to detect whether something was written with AI. I've had folks say they ran pangram on my writing and it was "100% AI". Except I don't use AI to write or edit anything I post publicly.
Can you give some examples of your writing that trigger "100% AI" on pangram? I'm very interested in studying pangram's false positives. Especially if it's something published before ~2024
I agree Pangram's UI is awful and often misleading. Their underlying classifier model is pretty accurate in my experience, though, at least in the sense that it has very few natural false positives. If you disagree, please send me some long-form verifiable false positives that were not explicitly written to trick Pangram. I love to learn.
I just sent you a long article which you could not have read in the time it took you to reply. I would suggest you start there and read that article, which outlines several cases where the author was able to create contrived false positives and negatives.
Contrived false positives and negatives could (and should!) always be possible. That doesn't tell us anything about the natural false positive and negative rates, and a lot of people who have actively tried to use voice instruction to get models to consistently fool the detector without iterating against it directly have failed.
Those people should just try posting a big quote from the article to HN, and then see their comment get instantly flagged due to HN's AI detection (which might as well be Pangram)
It's baffling to me how people are so dismissive of Pangram despite never using it, extrapolating their experience from GPTZero or something else.
Saying "more" seems generous. I don't trust Pangram at all, and I also don't trust the author at all. Both can be trustless charletans simultaneously, and I feel this speaks volumes about the state of digital media today.
So you would trust a known writers word over non deterministic software ? I read New Orleans will soon let AI handle 911 calls, Hope the AI believes the calls are from humans :)
All ML classifiers (and algorithms) have a non-zero false positive rate. Having an error rate is baked into every ML classifier and algorithm. And its always non-zero in practice. In fact, hitting every test in some sort of test suite is likely a sign of a less accurate classifier, not a more accurate one.
Can you give some examples of Pangram false positives? Ideally ones from before 2024, or otherwise ones from notable writers who started writing before 2024.
Three years ago, people were saying "LLMs just generate plausible-sounding text, they don't understand the notion of truth so they can't do verifiable work like math proofs."
They still can't. But very smart humans constructed ways to use the monkeys with typewriters (with a statistical advantage) to find correct answers to problems where they already knew how to verify the answer.
You need a problem where you both know what the solution looks like or can otherwise very quickly and efficiently determine that a solution is correct, but at the same time can't work out a correct solution with a similar amount of effort/time/cost as it took to determine how to verify a solution.
Most problems don't match that criteria. You usually either have a problem with a known method of solving, or you have a problem with no clear way of verifying the solution besides the act of finding the solution itself which would involve in some way proving it is correct, or you have a problem where verifying a solution takes a very long time or has a high cost or even can't be done more than once, so you need to try to determine the best solution without being able to actually test or verify.
Basically all problems just don't fit the "hard to solve but easy to verify" criteria to a degree that makes llms a good fit. On the other hand, there are so many problems that even a tiny fraction is a relatively large number.
No, he says it's a different category of problem, and ability to solve one doesn't carry over to the other and it doesn't imply intelligent understanding.
Right, which was true at the time. So hundreds of billions of dollars have been poured into making LLMs better at these tasks via pretraining, RL, RLHF, post training, etc. again all with something verifiable in the loop. In order to improve the thing in the loop, the loop itself needs to be verifiable.
There have only been a few thousand wars, and they’re all different and all different in the world in which they occurred. The dimensionality is absurd, which is not a problem for LLMs if there’s enough data, but in this case there isn’t.
> I don’t know that the lack of information should lead you to assume that the Mexico lawsuits are justified.
Gricean implicature.[1] The article wants to make the food companies look like the bad guys. If the lawsuits were clearly unjustified, the article could've made the food companies look worse by explaining the details. Since it didn't explain them, that means the lawsuits were probably justified.
reply