IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2406.02969.html
   My bibliography  Save this paper

Filtered not Mixed: Stochastic Filtering-Based Online Gating for Mixture of Large Language Models

Author

Listed:
  • Raeid Saqur
  • Anastasis Kratsios
  • Florian Krach
  • Yannick Limmer
  • Jacob-Junqi Tian
  • John Willes
  • Blanka Horvath
  • Frank Rudzicz

Abstract

We propose MoE-F -- a formalised mechanism for combining $N$ pre-trained expert Large Language Models (LLMs) in online time-series prediction tasks by adaptively forecasting the best weighting of LLM predictions at every time step. Our mechanism leverages the conditional information in each expert's running performance to forecast the best combination of LLMs for predicting the time series in its next step. Diverging from static (learned) Mixture of Experts (MoE) methods, MoE-F employs time-adaptive stochastic filtering techniques to combine experts. By framing the expert selection problem as a finite state-space, continuous-time Hidden Markov model (HMM), we can leverage the Wohman-Shiryaev filter. Our approach first constructs $N$ parallel filters corresponding to each of the $N$ individual LLMs. Each filter proposes its best combination of LLMs, given the information that they have access to. Subsequently, the $N$ filter outputs are aggregated to optimize a lower bound for the loss of the aggregated LLMs, which can be optimized in closed-form, thus generating our ensemble predictor. Our contributions here are: (I) the MoE-F algorithm -- deployable as a plug-and-play filtering harness, (II) theoretical optimality guarantees of the proposed filtering-based gating algorithm, and (III) empirical evaluation and ablative results using state of the art foundational and MoE LLMs on a real-world Financial Market Movement task where MoE-F attains a remarkable 17% absolute and 48.5% relative F1 measure improvement over the next best performing individual LLM expert.

Suggested Citation

  • Raeid Saqur & Anastasis Kratsios & Florian Krach & Yannick Limmer & Jacob-Junqi Tian & John Willes & Blanka Horvath & Frank Rudzicz, 2024. "Filtered not Mixed: Stochastic Filtering-Based Online Gating for Mixture of Large Language Models," Papers 2406.02969, arXiv.org.
  • Handle: RePEc:arx:papers:2406.02969
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2406.02969
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Viral V. Acharya & Matthew Richardson, 2012. "Implications of the Dodd-Frank Act," Annual Review of Financial Economics, Annual Reviews, vol. 4(1), pages 1-38, October.
    2. Raeid Saqur & Ken Kato & Nicholas Vinden & Frank Rudzicz, 2024. "NIFTY Financial News Headlines Dataset," Papers 2405.09747, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Reza Arabpour & John Armstrong & Luca Galimberti & Anastasis Kratsios & Giulia Livieri, 2024. "Low-dimensional approximations of the conditional law of Volterra processes: a non-positive curvature approach," Papers 2405.20094, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mohamed Drira & Muhammad Rashid, 2013. "Does A Size Limit Resolve Too Big To Fail Problems?," Accounting & Taxation, The Institute for Business and Finance Research, vol. 5(2), pages 65-77.
    2. Gamble, Edward & Caton, Gary & Aujogue, Kelig & Lee, Yen Teik, 2020. "Problems with crisis intervention: When the government wants to restrain big banks but punishes small businesses instead," Journal of Business Venturing Insights, Elsevier, vol. 14(C).
    3. Bernhard Kassner, 2023. "Taming Overconfident CEOs Through Stricter Financial Regulation," Rationality and Competition Discussion Paper Series 375, CRC TRR 190 Rationality and Competition.
    4. Kakhbod, Ali & Song, Fei, 2020. "Dynamic price discovery: Transparency vs. information design," Games and Economic Behavior, Elsevier, vol. 122(C), pages 203-232.
    5. Habib Ahmed & Ili Rahilah Ibrahim, 2018. "Financial Consumer Protection Regime in Malaysia: Assessment of the Legal and Regulatory Framework," Journal of Consumer Policy, Springer, vol. 41(2), pages 159-175, June.
    6. Chronopoulos, Dimitris K. & Wilson, John O.S. & Yilmaz, Muhammed H., 2023. "Regulatory oversight and bank risk," Journal of Financial Stability, Elsevier, vol. 64(C).
    7. M. Lundholm, 2021. "Compensation and Socio-Economic Status of Borrowers in Foreclosure: Evidence from Swedish Micro-data," Journal of Consumer Policy, Springer, vol. 44(1), pages 95-116, March.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2406.02969. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.