systemMessageComposition experimental configuration for more control over constructed system message

[ferenci84]

Description

Continuation of #4506

Replacement of #3787

On top of PR 4506 it adds experimental configuration systemMessageComposition. Changed documentation explains how it works.

Legacy logic remains working if user do not change the configuration.

Checklist

• The relevant docs, if any, have been updated or created
• The relevant tests, if any, have been updated or created

Screenshots

[ For visual changes, include screenshots. ]

Testing instructions

Experiment with config:

  "experimental": {
    "systemMessageComposition": "prepend"
  }

In the model description, you can also set:

     "promptTemplates": {
        "codeBlockInstructions": "My own instructions for code blocks",
        "toolUseRules": "My tool use rules"
      },

[netlify]

✅ Deploy Preview for continuedev canceled.

Name	Link
🔨 Latest commit	89329af
🔍 Latest deploy log	https://app.netlify.com/sites/continuedev/deploys/67c976d6cc11b3000807d045

[sammyvoncheese]

very cool change, I have been looking at doing something similar to customize/override all of the hidden system messages sent with each request.

[sestinj]

@ferenci84 I'm a bit concerned about all of the complexity that this adds. I'd like to get to some agreement about a plan that is easier to reason about before starting more work.

With something like the below you don't need to think about prepending, composing, etc.. It's a field that can be safely removed or altered at some point, and it seems to satisfy the needs of removing and replacing the default system prompt:

  - name: gpt-4o-mini
    provider: openai
    model: gpt-4o-mini
    apiKey: sk-xxx
    roles:
      - chat
      - edit
      - apply
    promptTemplates:
      defaultRules: ""

If defaultRules is set, then we would just entirely replace our defaults with this, which would effectively remove them if you set it to "". If you want to write your own tool use rules, then you can do so in the rules section as usual.

What do you think?

[ferenci84]

Do you want to remove systemMessage alltpgether? If not removed/deprecated, how would that work if any value is set?

[ferenci84]

If you want to write your own tool use rules, then you can do so in the rules section as usual

This is not clear to me. The tool use rule is added dynamically, only when tools are enabled. The rules section contains messages that are added always. Or, do I miss something?

Looking at the rules config option, it raises further questions:

rules:
  - uses: myprofile/my-mood-setter
    with:
      MOOD: happy
  - Always annotate Python functions with their parameter and return types
  - Always write Google style docstrings for functions and classes

Would it appear before or after the "defaultRules" specified in the model, or if "defaultRules" are defined, it replaces everything? What is the advantage of having a list of rules over having a single systemMessage?

(It's a side question, but not sure whether the "uses/with" combination is documented anywhere?)

[ferenci84]

@sestinj I had an idea yesterday, that may be simple to average users, but would provide additional tools for power user, let me explain:

I was reading couple of weeks ago in Claude docs that they recommend using the system message for role prompting, and put any other instructions (like how to format code) in the user message instead.

My idea is to have have promptTemplates with the keys defaultRules and toolUseRules, but also systemMessageTemplate AND userMessageTemplate, both with placeholders using the same keys as in promptTemplates, and with default already set for the user (it could be just systemMessage if we want to avoid changing the old name, but if there may be placeholders, template may be more appropriate).

This would have the following advantages:

• Average or new users can change the defaultRules and toolUseRules without having to understand them more (but as they are not editing a config named "systemMessage" they may not be surprised so much that something is appended to their inputted text)
• Users who want control over the whole systemMessage, can just put their own text and nothing will be added to it, since if there is no placeholder found, there will be no replacement, they work as simple string
• Users who actually read the deep-dive section of the documentation, can alter the systemMessageTemplate and userMessageTemplate so that they can add/remove parts
• Specifically for Claude, it would be possible to add the a standard defultRules and the toolUseRules (that would be added only when the user actually enables any of the tools) right within the user message and keep the system message contain only the role.

The following example elaborates what I thought about:

Simple user-defined default rules, default toolUseRules would be appended:

  - name: gpt-4o-mini
    provider: openai
    model: gpt-4o-mini
    promptTemplates:
      defaultRules: "Custom prompt"
      
 - name: gpt-4o-mini-1
    provider: openai
    model: gpt-4o-mini
    promptTemplates:
      defaultRules: "Custom prompt"
      toolUseRules: "Custom tool use rules"

User may overwrite the whole system message:

  - name: gpt-4o-mini
    provider: openai
    model: gpt-4o-mini
    systemMessageTemplate: ""

  - name: gpt-4o-mini-1
    provider: openai
    model: gpt-4o-mini
    systemMessageTemplate: "You are an expert developer"

# this has same effect as setting both prompt templates:
 - name: gpt-4o-mini
    provider: openai
    model: gpt-4o-mini
    promptTemplates:
      defaultRules: ""
      toolUseRules: ""
      
 - name: gpt-4o-mini-1
    provider: openai
    model: gpt-4o-mini
    promptTemplates:
      defaultRules: "You are an expert developer"
      toolUseRules: ""

When user make edit in the config, these defaults would appear (Note: this is a draft, we can talk about the best format.):

 - name: gpt-4o-mini-1
    provider: openai
    model: gpt-4o-mini
    systemMessageTemplate: "{defaultRules}{toolUseRules}"
    userMessageTemplate: "{context}{userMessage}"
    promptTemplates:
      defaultRules: "Always include the language and file name in the info string when you write code blocks, for example '```python file.py'."
      toolUseRules: "When using tools, follow the following guidelines:\n- Avoid calling tools unless they are absolutely necessary. For example, if you are asked a simple programming question you do not need web search. As another example, if the user asks you to explain something about code, do not create a new file."

For specific models, the placement of "rules" may be changed (we may also allow arbitrary keys in promptTemplates as demonstrated here):

 - name: anthropic-claude
    provider: anthropic
    model: anthropic-claude-3.7
    systemMessageTemplate: "{role}"
    userMessageTemplate: "{context}{userMessage}{defaultRules}{toolUseRules}"
    promptTemplates:
      role: "You are an expert developer"
      defaultRules: "Always include the language and file name in the info string when you write code blocks, for example '```python file.py'."
      toolUseRules: "When using tools, follow the following guidelines:\n- Avoid calling tools unless they are absolutely necessary. For example, if you are asked a simple programming question you do not need web search. As another example, if the user asks you to explain something about code, do not create a new file."

Additional cool features may also be added:

• A top-level promptTemplates config may let users reuse messages, and use those keys in more than one model setup.
• For some models, systemMessageTemplate and userMessageTemplate may be used in the model defaults, for example we can set default empty system message for Deepseek R1 as recommended by their readme, and put the default and tool use rules in the user message for Claude models.
• There may be rules that could be put only in the first user message instead of repeating it in each. For example if I instruct Claude the output the whole edited code in one of the messages, it will behave the same way in every subsequent messages. (Btw if I do the same in system message, it simply doesn't follow the instructions)
• The method could be reused with just a few lines of code, for example if o1 requires providing developer message instead of system message.

The placeholder part is already done and working, and there is a smart handling of whitespaces before/after the replaced parts (this complexity is clearly encapsulated in a single function). Also with the change in this PR, the final system message is constructed in a single place, so we can put everything about that into a single file and make changes in that single place, instead of having to change several places in the code-base.

Let me know what do you think of it as direction and if you have any idea. I don't think it should be done in one-shot in a full-featured, single change, rather it should be broken down to multiple smaller changes, maybe first doing changes only about the system message (most parts are already done for this), then, if working well, working on the user message template (maybe after involving other users into the talk), and then adding extra features described above and possibly implementing sensible defaults for specific models.

[sammyvoncheese]

I have wanted to implement something like this myself, specifically to overwrite all continue system messages, and tooling messages by model. I am willing to help test if needed.

…

On Mon, Mar 17, 2025 at 6:48 AM ferenci84 ***@***.***> wrote: @sestinj <https://github.com/sestinj> I had an idea yesterday, that may be simple to average users, but would provide additional tools for power user, let me explain: I was reading couple of weeks ago in Claude docs that they recommend using the system message for role prompting, and put any other instructions (like how to format code) in the user message instead. My idea is to have have promptTemplates with the keys defaultRules and toolUseRules, but also systemMessageTemplate AND userMessageTemplate, both with placeholders using the same keys as in promptTemplates, and with default already set for the user (it could be just systemMessage if we want to avoid changing the old name, but if there may be placeholders, template may be more appropriate). This would have the following advantages: - Average or new users can change the defaultRules and toolUseRules without having to understand them more (but as they are not editing a config named "systemMessage" they may not be surprised so much that something is appended to their inputted text) - Users who want control over the whole systemMessage, can just put their own text and nothing will be added to it, since if there is no placeholder found, there will be no replacement, they work as simple string - Users who actually read the deep-dive section of the documentation, can alter the systemMessageTemplate and userMessageTemplate so that they can add/remove parts - Specifically for Claude, it would be possible to add the a standard defultRules and the toolUseRules (that would be added only when the user actually enables any of the tools) right within the user message and keep the system message contain only the role. The following example elaborates what I thought about: Simple user-defined default rules, default toolUseRules would be appended: - name: gpt-4o-mini provider: openai model: gpt-4o-mini promptTemplates: defaultRules: "Custom prompt" - name: gpt-4o-mini-1 provider: openai model: gpt-4o-mini promptTemplates: defaultRules: "Custom prompt" toolUseRules: "Custom tool use rules" User may overwrite the whole system message: - name: gpt-4o-mini provider: openai model: gpt-4o-mini systemMessageTemplate: "" - name: gpt-4o-mini-1 provider: openai model: gpt-4o-mini systemMessageTemplate: "You are an expert developer" # this has same effect as setting both prompt templates: - name: gpt-4o-mini provider: openai model: gpt-4o-mini promptTemplates: defaultRules: "" toolUseRules: "" - name: gpt-4o-mini-1 provider: openai model: gpt-4o-mini promptTemplates: defaultRules: "You are an expert developer" toolUseRules: "" When user make edit in the config, these defaults would appear (Note: this is a draft, we can talk about the best format.): - name: gpt-4o-mini-1 provider: openai model: gpt-4o-mini systemMessageTemplate: "{defaultRules}{toolUseRules}" userMessageTemplate: "{context}{userMessage}" promptTemplates: defaultRules: "Always include the language and file name in the info string when you write code blocks, for example '```python file.py'." toolUseRules: "When using tools, follow the following guidelines:\n- Avoid calling tools unless they are absolutely necessary. For example, if you are asked a simple programming question you do not need web search. As another example, if the user asks you to explain something about code, do not create a new file." For specific models, the placement of "rules" may be changed (we may also allow arbitrary keys in promptTemplates as demonstrated here): - name: anthropic-claude provider: anthropic model: anthropic-claude-3.7 systemMessageTemplate: "{role}" userMessageTemplate: "{context}{userMessage}{defaultRules}{toolUseRules}" promptTemplates: role: "You are an expert developer" defaultRules: "Always include the language and file name in the info string when you write code blocks, for example '```python file.py'." toolUseRules: "When using tools, follow the following guidelines:\n- Avoid calling tools unless they are absolutely necessary. For example, if you are asked a simple programming question you do not need web search. As another example, if the user asks you to explain something about code, do not create a new file." Additional cool features may also be added: - A top-level promptTemplates config may let users reuse messages, and use those keys in more than one model setup. - For some models, systemMessageTemplate and userMessageTemplate may be used in the model defaults, for example we can set default empty system message for Deepseek R1 as recommended by their readme, and put the default and tool use rules in the user message for Claude models. - There may be rules that could be put only in the first user message instead of repeating it in each. For example if I instruct Claude the output the whole edited code in one of the messages, it will behave the same way in every subsequent messages. (Btw if I do the same in system message, it simply doesn't follow the instructions) - The method could be reused with just a few lines of code, for example if o1 requires providing developer message instead of system message. The placeholder part is already done and working, and there is a smart handling of whitespaces before/after the replaced parts (this complexity is clearly encapsulated in a single function). Also with the change in this PR, the final system message is constructed in a single place, so we can put everything about that into a single file and make changes in that single place, instead of having to change several places in the code-base. Let me know what do you think of it as direction and if you have any idea. I don't think it should be done in one-shot in a full-featured, single change, rather it should be broken down to multiple smaller changes, maybe first doing changes only about the system message (most parts are already done for this), then, if working well, working on the user message template (maybe after involving other users into the talk), and then adding extra features described above and possibly implementing sensible defaults for specific models. — Reply to this email directly, view it on GitHub <#4507 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF2VHLTQUNDMKLKT43ROVJD2U2R6DAVCNFSM6AAAAABYOALJMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRZGA2DEMBYG4> . You are receiving this because you commented.Message ID: ***@***.***> [image: ferenci84]*ferenci84* left a comment (continuedev/continue#4507) <#4507 (comment)> @sestinj <https://github.com/sestinj> I had an idea yesterday, that may be simple to average users, but would provide additional tools for power user, let me explain: I was reading couple of weeks ago in Claude docs that they recommend using the system message for role prompting, and put any other instructions (like how to format code) in the user message instead. My idea is to have have promptTemplates with the keys defaultRules and toolUseRules, but also systemMessageTemplate AND userMessageTemplate, both with placeholders using the same keys as in promptTemplates, and with default already set for the user (it could be just systemMessage if we want to avoid changing the old name, but if there may be placeholders, template may be more appropriate). This would have the following advantages: - Average or new users can change the defaultRules and toolUseRules without having to understand them more (but as they are not editing a config named "systemMessage" they may not be surprised so much that something is appended to their inputted text) - Users who want control over the whole systemMessage, can just put their own text and nothing will be added to it, since if there is no placeholder found, there will be no replacement, they work as simple string - Users who actually read the deep-dive section of the documentation, can alter the systemMessageTemplate and userMessageTemplate so that they can add/remove parts - Specifically for Claude, it would be possible to add the a standard defultRules and the toolUseRules (that would be added only when the user actually enables any of the tools) right within the user message and keep the system message contain only the role. The following example elaborates what I thought about: Simple user-defined default rules, default toolUseRules would be appended: - name: gpt-4o-mini provider: openai model: gpt-4o-mini promptTemplates: defaultRules: "Custom prompt" - name: gpt-4o-mini-1 provider: openai model: gpt-4o-mini promptTemplates: defaultRules: "Custom prompt" toolUseRules: "Custom tool use rules" User may overwrite the whole system message: - name: gpt-4o-mini provider: openai model: gpt-4o-mini systemMessageTemplate: "" - name: gpt-4o-mini-1 provider: openai model: gpt-4o-mini systemMessageTemplate: "You are an expert developer" # this has same effect as setting both prompt templates: - name: gpt-4o-mini provider: openai model: gpt-4o-mini promptTemplates: defaultRules: "" toolUseRules: "" - name: gpt-4o-mini-1 provider: openai model: gpt-4o-mini promptTemplates: defaultRules: "You are an expert developer" toolUseRules: "" When user make edit in the config, these defaults would appear (Note: this is a draft, we can talk about the best format.): - name: gpt-4o-mini-1 provider: openai model: gpt-4o-mini systemMessageTemplate: "{defaultRules}{toolUseRules}" userMessageTemplate: "{context}{userMessage}" promptTemplates: defaultRules: "Always include the language and file name in the info string when you write code blocks, for example '```python file.py'." toolUseRules: "When using tools, follow the following guidelines:\n- Avoid calling tools unless they are absolutely necessary. For example, if you are asked a simple programming question you do not need web search. As another example, if the user asks you to explain something about code, do not create a new file." For specific models, the placement of "rules" may be changed (we may also allow arbitrary keys in promptTemplates as demonstrated here): - name: anthropic-claude provider: anthropic model: anthropic-claude-3.7 systemMessageTemplate: "{role}" userMessageTemplate: "{context}{userMessage}{defaultRules}{toolUseRules}" promptTemplates: role: "You are an expert developer" defaultRules: "Always include the language and file name in the info string when you write code blocks, for example '```python file.py'." toolUseRules: "When using tools, follow the following guidelines:\n- Avoid calling tools unless they are absolutely necessary. For example, if you are asked a simple programming question you do not need web search. As another example, if the user asks you to explain something about code, do not create a new file." Additional cool features may also be added: - A top-level promptTemplates config may let users reuse messages, and use those keys in more than one model setup. - For some models, systemMessageTemplate and userMessageTemplate may be used in the model defaults, for example we can set default empty system message for Deepseek R1 as recommended by their readme, and put the default and tool use rules in the user message for Claude models. - There may be rules that could be put only in the first user message instead of repeating it in each. For example if I instruct Claude the output the whole edited code in one of the messages, it will behave the same way in every subsequent messages. (Btw if I do the same in system message, it simply doesn't follow the instructions) - The method could be reused with just a few lines of code, for example if o1 requires providing developer message instead of system message. The placeholder part is already done and working, and there is a smart handling of whitespaces before/after the replaced parts (this complexity is clearly encapsulated in a single function). Also with the change in this PR, the final system message is constructed in a single place, so we can put everything about that into a single file and make changes in that single place, instead of having to change several places in the code-base. Let me know what do you think of it as direction and if you have any idea. I don't think it should be done in one-shot in a full-featured, single change, rather it should be broken down to multiple smaller changes, maybe first doing changes only about the system message (most parts are already done for this), then, if working well, working on the user message template (maybe after involving other users into the talk), and then adding extra features described above and possibly implementing sensible defaults for specific models. — Reply to this email directly, view it on GitHub <#4507 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF2VHLTQUNDMKLKT43ROVJD2U2R6DAVCNFSM6AAAAABYOALJMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMRZGA2DEMBYG4> . You are receiving this because you commented.Message ID: ***@***.***>