GPT-4 was exposed as "stupid"! In order to reduce costs, OpenAI secretly engaged in "small moves"?

Organize | Zheng Liyuan

Listing | CSDN (ID: CSDNnews)

“Do you feel that the quality of GPT-4 has declined recently?”

As early as mid-May this year, GPT-4 users posted in OpenAI’s online developer forum, saying that GPT-4 seemed to be “becoming stupid”:

"I’ve been using ChatGPT for a while, and I’ve been a GPT Plus user since the release of GPT-4. I generally use GPT-4 to help with analysis and creation of long-form content. In the past, GPT-4 seemed to work well understood my request. But now, it seems to lose track of the information, give me a lot of wrong information, and misinterpret my request more and more… Currently, GPT-4 feels more like GPT-3.5.

Has anyone else noticed this, or is it just me who stumbled across this issue? "

As it turns out, the GPT-4 user was not alone—a number of users commented on the thread: “I feel the same way!”

The problematic new version of GPT-4

According to the feedback of many users, GPT-4 has not only declined in the analysis and creation of long-form content, but also in the quality of writing.

Peter Yang, head of product at Roblox, tweeted that the output of the GPT-4 model is faster, but the quality has deteriorated: "Just simple problems, such as making writing clearer and concise and providing ideas…in my opinion , the writing quality has declined.”

The logic ability of the latest version of GPT-4 drops significantly when discussing/evaluating complex inverse problems, different rates or patterns of change, and spatio-temporal variability.

“Before the GPT-4 update, I rarely got error responses, but now I have to double-check all output (i.e. double negative conditions are now sometimes not properly converted to positive conditions). I think these errors are more similar to GPT -3.5 instead of the previous GPT-4 inference level.”

Even in terms of encoding ability, the output quality of the new version of GPT-4 is not as good as before.

A developer using GPT-4 to write functional code for a website complained: “The current GPT-4 is very disappointing. It’s like driving a Ferrari for a month, and then suddenly it becomes an old pickup truck. Not sure I would want to keep paying for it.”

Another developer also mentioned that GPT-4 now loops out code: “Totally sucks, GPT-4 starts looping out code or other information over and over again. I let it write code, and it wrote at a point , suddenly a “”, and then start again! Compared with before, it is an idiot now.”

In addition, in Twitter and OpenAI’s online developer forums, users continue to report that the new version of GPT-4 has weakened logic, generated many error responses, failed to track the information provided, did not follow the instructions, and forgot to write in the basic software code. Putting parentheses in , remembering only the most recent reminders, and more.

Regarding the performance of GPT-4’s sudden “reduction of intelligence”, some users speculated: “The current version feels very different from the version when it was just launched. I guess OpenAI chose to compromise on quality in order to accommodate more customers!”

From this point of view, the evaluation of GPT-4 by users today is indeed not as good as the peak period of “wind evaluation” when it first debuted.

GPT-4 is faster, but also “stupid”

At the end of last year, ChatGPT based on GPT-3.5 was born, and its excellent generation ability set off an AIGC boom. Therefore, when OpenAI announced GPT-4, which is more powerful than GPT-3.5, in March this year, the whole world was amazed.

At that time, GPT-4 was called “the most powerful AI model in history”, especially its multi-modality, which means that it can understand both images and text input, so it quickly became a popular tool for developers and other technology industry. The model of choice for professionals has also produced more praise for GPT-4: generating a website in 10 seconds, passing the most difficult American law test, and passing the MIT undergraduate mathematics test with full marks…

However, when people are amazed at the power of GPT-4, many people are also shocked by its cost and response speed. “GPT-4 is slow, but very accurate,” said Sharon Zhou, CEO of Lamini, a startup that helps developers build custom large-scale language models.

Until May, GPT-4 remained “slow and expensive but accurate”—later, GPT-4 responded faster, and at the same time users questioned its performance degradation.

For this phenomenon, several AI experts, including Sharon Zhou, believe that OpenAI may be creating several smaller GPT-4 models that function similarly to the larger models but are less expensive to run.

Experts speculate: it may be related to MoE technology

According to Sharon Zhou’s introduction, this method is called Mixture-of-Experts (MoE), that is, a mixed expert system. MoE technology is an integrated learning technology developed in the field of neural networks, and it is also a key technology for training models with trillions of parameters. Due to the increasing size of the model at this stage, the training overhead is also increasing, and MoE The technology can dynamically activate part of the neural network, thereby greatly increasing the amount of model parameters without increasing the amount of calculation.

Specifically, MoE decomposes the predictive modeling task into several subtasks, trains an expert model (Expert Model) on each subtask, and develops a gating model (Gating Model), which can be predicted according to the input Come learn which experts to trust, and combine forecast results.

So what is the situation when MoE technology is referenced to GPT-4? Sharon Zhou explained that in GPT-4, these small expert models will be trained for different tasks and subject areas. For example, there can be small GPT-4 expert models for biology, physics, chemistry, etc. 4 When a question is asked, the new system knows which expert model to send the question to. Also, just in case, the new system might send queries to two or more expert models and then mash the results together.

For this approach, Sharon Zhou described it as “The Ship of Theseus” (a paradox about identity replacement, assuming that the constituent elements of an object are replaced, but is it still the original object?), that is, with Over time, OpenAI will replace parts of GPT-4: “OpenAI is turning GPT-4 into a small fleet.”

Based on the above speculation, Sharon Zhou believes that GPT-4’s recent “stupid” remarks are likely to be related to the MoE training method: “When users test GPT-4, we will ask many different questions, and the scale is small. The GPT-4 expert model isn’t going to do that well, but it’s collecting our data, and it’s improving and learning.”

**GPT-4 architecture exposed? **

Several AI experts also released so-called “GPT-4 architecture details” this week amid growing user feedback about GPT-4’s “goofiness.”

Among them, a Twitter blogger named Yam Peleg said that GPT-4 has about 1.8 trillion parameters, spans 120 layers, is more than 10 times larger than GPT-3, and is trained on about 13T tokens. The training cost About $63 million… It is worth mentioning that Yam Peleg also said that OpenAI is using MoE, which is to reduce the cost of GPT-4 operation by using 16 mixed expert models.

As of now, OpenAI has not responded to this statement. But Oren Etzioni, the founding CEO of the Allen Institute for Artificial Intelligence, told the media: “Although I have not been confirmed, I think these speculations should be roughly correct.”

He explained that there are generally two reasons for using the MOE method: either you want to generate a better response, or you want a cheaper, faster response.

“Ideally, MOE will allow you to gain both advantages at the same time, but in reality, you usually need to make a trade-off between cost and quality.” Based on this, Oren Etzioni believes that combined with the current situation, OpenAI seems to reduce GPT -4 at the cost of sacrificing some quality.

So what is your opinion on this matter?

Reference link: