From f31e17030c32ec66c4fb8446ff0c3839e5638666 Mon Sep 17 00:00:00 2001
From: Jax922 <1322037892@qq.com>
Date: Tue, 13 May 2025 15:05:56 +0800
Subject: [PATCH] update
---
CODE_OF_CONDUCT.md | 128 --
README.md | 199 ---
README_en.md | 1509 -----------------
images/1-wiki.png | Bin 139475 -> 0 bytes
images/2-wiki.png | Bin 74677 -> 0 bytes
images/3-wiki.png | Bin 235326 -> 0 bytes
images/4-wiki.png | Bin 106870 -> 0 bytes
images/5-wiki.png | Bin 245144 -> 0 bytes
images/LLM-structure-moe.png | Bin 123671 -> 0 bytes
images/LLM-structure.png | Bin 380409 -> 0 bytes
images/and_huggingface.png | Bin 182102 -> 0 bytes
images/and_modelscope.png | Bin 153731 -> 0 bytes
images/compare_radar.png | Bin 531337 -> 0 bytes
images/dataset.jpg | Bin 149354 -> 0 bytes
images/gpt3_config.png | Bin 67201 -> 0 bytes
images/logo.png | Bin 507055 -> 0 bytes
images/logo2.png | Bin 629688 -> 0 bytes
images/minimind2.gif | Bin 3976646 -> 0 bytes
images/pre_512_loss.png | Bin 572682 -> 0 bytes
images/pre_768_loss.png | Bin 544089 -> 0 bytes
images/sft_512_loss.png | Bin 1030620 -> 0 bytes
images/sft_768_loss.png | Bin 966016 -> 0 bytes
model/LMConfig.py | 6 +
model/model.py | 8 +-
requirements.txt | 141 +-
run_file/DynamicKV-LLM_Mini_Minimind.sh | 48 +
.../DynamicKV-LLM_Small_Minimind.sh | 0
train_pretrain_accelerate.py | 6 +-
28 files changed, 192 insertions(+), 1853 deletions(-)
delete mode 100644 CODE_OF_CONDUCT.md
delete mode 100644 README.md
delete mode 100644 README_en.md
delete mode 100644 images/1-wiki.png
delete mode 100644 images/2-wiki.png
delete mode 100644 images/3-wiki.png
delete mode 100644 images/4-wiki.png
delete mode 100644 images/5-wiki.png
delete mode 100644 images/LLM-structure-moe.png
delete mode 100755 images/LLM-structure.png
delete mode 100644 images/and_huggingface.png
delete mode 100644 images/and_modelscope.png
delete mode 100644 images/compare_radar.png
delete mode 100644 images/dataset.jpg
delete mode 100644 images/gpt3_config.png
delete mode 100644 images/logo.png
delete mode 100644 images/logo2.png
delete mode 100644 images/minimind2.gif
delete mode 100644 images/pre_512_loss.png
delete mode 100644 images/pre_768_loss.png
delete mode 100644 images/sft_512_loss.png
delete mode 100644 images/sft_768_loss.png
create mode 100644 run_file/DynamicKV-LLM_Mini_Minimind.sh
rename run_accelerate.sh => run_file/DynamicKV-LLM_Small_Minimind.sh (100%)
diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
deleted file mode 100644
index 18c9147..0000000
--- a/CODE_OF_CONDUCT.md
+++ /dev/null
@@ -1,128 +0,0 @@
-# Contributor Covenant Code of Conduct
-
-## Our Pledge
-
-We as members, contributors, and leaders pledge to make participation in our
-community a harassment-free experience for everyone, regardless of age, body
-size, visible or invisible disability, ethnicity, sex characteristics, gender
-identity and expression, level of experience, education, socio-economic status,
-nationality, personal appearance, race, religion, or sexual identity
-and orientation.
-
-We pledge to act and interact in ways that contribute to an open, welcoming,
-diverse, inclusive, and healthy community.
-
-## Our Standards
-
-Examples of behavior that contributes to a positive environment for our
-community include:
-
-* Demonstrating empathy and kindness toward other people
-* Being respectful of differing opinions, viewpoints, and experiences
-* Giving and gracefully accepting constructive feedback
-* Accepting responsibility and apologizing to those affected by our mistakes,
- and learning from the experience
-* Focusing on what is best not just for us as individuals, but for the
- overall community
-
-Examples of unacceptable behavior include:
-
-* The use of sexualized language or imagery, and sexual attention or
- advances of any kind
-* Trolling, insulting or derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or email
- address, without their explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
- professional setting
-
-## Enforcement Responsibilities
-
-Community leaders are responsible for clarifying and enforcing our standards of
-acceptable behavior and will take appropriate and fair corrective action in
-response to any behavior that they deem inappropriate, threatening, offensive,
-or harmful.
-
-Community leaders have the right and responsibility to remove, edit, or reject
-comments, commits, code, wiki edits, issues, and other contributions that are
-not aligned to this Code of Conduct, and will communicate reasons for moderation
-decisions when appropriate.
-
-## Scope
-
-This Code of Conduct applies within all community spaces, and also applies when
-an individual is officially representing the community in public spaces.
-Examples of representing our community include using an official e-mail address,
-posting via an official social media account, or acting as an appointed
-representative at an online or offline event.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported to the community leaders responsible for enforcement at
-.
-All complaints will be reviewed and investigated promptly and fairly.
-
-All community leaders are obligated to respect the privacy and security of the
-reporter of any incident.
-
-## Enforcement Guidelines
-
-Community leaders will follow these Community Impact Guidelines in determining
-the consequences for any action they deem in violation of this Code of Conduct:
-
-### 1. Correction
-
-**Community Impact**: Use of inappropriate language or other behavior deemed
-unprofessional or unwelcome in the community.
-
-**Consequence**: A private, written warning from community leaders, providing
-clarity around the nature of the violation and an explanation of why the
-behavior was inappropriate. A public apology may be requested.
-
-### 2. Warning
-
-**Community Impact**: A violation through a single incident or series
-of actions.
-
-**Consequence**: A warning with consequences for continued behavior. No
-interaction with the people involved, including unsolicited interaction with
-those enforcing the Code of Conduct, for a specified period of time. This
-includes avoiding interactions in community spaces as well as external channels
-like social media. Violating these terms may lead to a temporary or
-permanent ban.
-
-### 3. Temporary Ban
-
-**Community Impact**: A serious violation of community standards, including
-sustained inappropriate behavior.
-
-**Consequence**: A temporary ban from any sort of interaction or public
-communication with the community for a specified period of time. No public or
-private interaction with the people involved, including unsolicited interaction
-with those enforcing the Code of Conduct, is allowed during this period.
-Violating these terms may lead to a permanent ban.
-
-### 4. Permanent Ban
-
-**Community Impact**: Demonstrating a pattern of violation of community
-standards, including sustained inappropriate behavior, harassment of an
-individual, or aggression toward or disparagement of classes of individuals.
-
-**Consequence**: A permanent ban from any sort of public interaction within
-the community.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage],
-version 2.0, available at
-https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
-
-Community Impact Guidelines were inspired by [Mozilla's code of conduct
-enforcement ladder](https://github.com/mozilla/diversity).
-
-[homepage]: https://www.contributor-covenant.org
-
-For answers to common questions about this code of conduct, see the FAQ at
-https://www.contributor-covenant.org/faq. Translations are available at
-https://www.contributor-covenant.org/translations.
diff --git a/README.md b/README.md
deleted file mode 100644
index 7ddb33e..0000000
--- a/README.md
+++ /dev/null
@@ -1,199 +0,0 @@
-
-
-* This open-source project aims to train a super-small language model **MiniMind** with only 3 RMB cost and 2 hours,
- starting completely from scratch.
-* The **MiniMind** series is extremely lightweight, with the smallest version being $\frac{1}{7000}$ the size of GPT-3,
- making it possible to train quickly on even the most ordinary personal GPUs.
-* The project also open-sources the minimalist structure of the large model, including extensions for shared mixed
- experts (MoE), dataset cleaning, pretraining, supervised fine-tuning (SFT), LoRA fine-tuning, direct preference
- optimization (DPO) algorithms, and model distillation algorithms, along with the full code of the process.
-* **MiniMind** also expands into vision multimodal VLM: [MiniMind-V](https://github.com/jingyaogong/minimind-v).
-* All core algorithm code is reconstructed from scratch using native PyTorch! It does not rely on abstract interfaces
- provided by third-party libraries.
-* This is not only a full-stage open-source reproduction of a large language model but also a tutorial for beginners in
- LLM.
-* We hope this project will serve as an inspiring example for everyone, helping to enjoy the fun of creation and
- promoting the progress of the wider AI community!
-
- > To avoid misunderstanding, the "2 hours" test is based on NVIDIA 3090 hardware (single GPU), and the "3 RMB" refers
- to the GPU server rental cost. Details of the specifications can be found below.
-
----
-
-
-
-# 📌 Introduction
-
-The emergence of Large Language Models (LLMs) has sparked unprecedented global attention on AI. Whether it's ChatGPT,
-DeepSeek, or Qwen, their stunning performance leaves people in awe. However, the massive scale of hundreds of billions
-of parameters makes it not only difficult to train them on personal devices, but also almost impossible to deploy them.
-Opening the "black box" of large models and exploring their inner workings is exhilarating! Sadly, 99% of explorations
-can only stop at fine-tuning existing large models with techniques like LoRA, learning a few new commands or tasks. It's
-like teaching Newton how to use a 21st-century smartphone—though interesting, it completely deviates from the original
-goal of understanding the essence of physics. Meanwhile, third-party large model frameworks and toolkits, such as
-transformers+trl, almost only expose highly abstract interfaces. With just 10 lines of code, you can complete the entire
-training process of "loading model + loading dataset + inference + reinforcement learning". While this efficient
-encapsulation is convenient, it's like a high-speed spaceship, isolating us from the underlying implementation and
-hindering our opportunity to dive deep into the core code of LLMs. However, "building a plane with Legos is far more
-exciting than flying first-class!" What's worse, the internet is flooded with paid courses and marketing accounts,
-selling AI tutorials with flawed and half-understood content. Therefore, the goal of this project is to lower the
-learning threshold for LLMs, allowing everyone to start by understanding each line of code, and to train a very small
-language model from scratch, not just performing **inference**! With server costs of less than 3 RMB, you can experience
-the entire process of building a language model from 0 to 1. Let's enjoy the fun of creation together!
-
-> [!NOTE]
-> (As of 2025-02-07) The MiniMind series has completed pretraining for multiple models, with the smallest one being only
-> 25.8M (0.02B) and capable of smooth conversation!
-
-
-Models List
-
-| Model (Size) | Inference Usage (Approx.) | Release |
-|-------------------------|---------------------------|------------|
-| MiniMind2-small (26M) | 0.5 GB | 2025.02.06 |
-| MiniMind2-MoE (145M) | 1.0 GB | 2025.02.06 |
-| MiniMind2 (104M) | 1.0 GB | 2025.02.06 |
-| minimind-v1-small (26M) | 0.5 GB | 2024.08.28 |
-| minimind-v1-moe (4×26M) | 1.0 GB | 2024.09.17 |
-| minimind-v1 (108M) | 1.0 GB | 2024.09.01 |
-
-
-
-**Project Includes**
-
-- All code for the MiniMind-LLM structure (Dense+MoE models).
-- Detailed training code for the Tokenizer.
-- Full training code for Pretrain, SFT, LoRA, RLHF-DPO, and model distillation.
-- High-quality datasets collected, distilled, cleaned, and deduplicated at all stages, all open-source.
-- From scratch implementation of pretraining, instruction fine-tuning, LoRA, DPO reinforcement learning, and white-box
- model distillation. Most key algorithms do not rely on third-party encapsulated frameworks and are all open-source.
-- Compatible with third-party frameworks like `transformers`, `trl`, `peft`, etc.
-- Training supports single machine single GPU, single machine multi-GPU (DDP, DeepSpeed), and wandb visualized training
- processes. Supports dynamic start/stop of training.
-- Model testing on third-party evaluation benchmarks (C-Eval, C-MMLU, OpenBookQA, etc.).
-- A minimal server implementing the Openai-Api protocol, easy to integrate into third-party ChatUI applications (
- FastGPT, Open-WebUI, etc.).
-- A simple chat WebUI front-end implemented using streamlit.
-- Reproduction (distillation/RL) of the large inference model DeepSeek-R1 as the MiniMind-Reason model, **data + model**
- all open-source!
-
-We hope this open-source project can help LLM beginners quickly get started!
-
-### 👉**Update log**
-
-
-2025-02-09 (newest 🎉🎉🎉)
-
-- Major update since the release, with the release of MiniMind2 Series.
-- Almost all code has been refactored, using a more streamlined and unified structure.
- For compatibility with old code, please refer to
- the [🔗Old Repository Contents🔗](https://github.com/jingyaogong/minimind/tree/6e9cd28ef9b34a0a10afbdf6f59e65cb6e628efb).
-- Removed the data preprocessing step. Unified dataset format, switched to `jsonl` format to eliminate issues with
- dataset downloads.
-- MiniMind2 series shows a significant improvement over MiniMind-V1.
-- Minor issues: {kv-cache syntax is more standard, MoE load balancing loss is considered, etc.}
-- Provided a training solution for transferring the model to private datasets (e.g., medical models, self-awareness
- examples).
-- Streamlined the pretraining dataset and significantly improved the quality of the pretraining data, greatly reducing
- the time needed for personal rapid training, with a single 3090 GPU achieving reproduction in just 2 hours!
-- Updated: LoRA fine-tuning now operates outside of the `peft` wrapper, implemented LoRA process from scratch; DPO
- algorithm is implemented using native PyTorch; native model white-box distillation.
-- MiniMind2-DeepSeek-R1 series distilled models have been created!
-- MiniMind2 now has some English proficiency!
-- Updated MiniMind2 performance results based on additional large model benchmark tests.
-
-
-
-
-2024-10-05
-
-- Expanded MiniMind to include multimodal capabilities—visual.
-- Check out the twin project [minimind-v](https://github.com/jingyaogong/minimind-v) for more details!
-
-
-
-
-2024-09-27
-
-- Updated preprocessing method for the pretrain dataset on 09-27 to ensure text integrity. The method of converting to
- .bin for training has been abandoned (slightly sacrificing training speed).
-- The preprocessed pretrain file is now named: pretrain_data.csv.
-- Removed some redundant code.
-
-
-
-
-2024-09-17
-
-- Updated minimind-v1-moe model.
-- To avoid ambiguity, the mistral_tokenizer is no longer used, and all tokenization is done with the custom
- minimind_tokenizer.
-
-
-
-
-2024-09-01
-
-- Updated minimind-v1 (108M) model, using minimind_tokenizer, with 3 pretraining rounds + 10 SFT rounds, allowing for
- more comprehensive training and improved performance.
-- The project has been deployed on ModelScope Creative Space and can be experienced on the site:
-- [🔗ModelScope Online Experience🔗](https://www.modelscope.cn/studios/gongjy/minimind)
-
-
-
-
-2024-08-27
-
-- Initial open-source release of the project.
-
-
-
-# 📌 Quick Start
-
-
-Sharing My Hardware and Software Configuration (For Reference Only)
-
-* CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
-* RAM: 128 GB
-* GPU: NVIDIA GeForce RTX 3090(24GB) * 8
-* Ubuntu==20.04
-* CUDA==12.2
-* Python==3.10.16
-* [requirements.txt](./requirements.txt)
-
-
-
-### Step 0
-
-```bash
-git clone https://github.com/jingyaogong/minimind.git
-```
-
-## Ⅰ Test Pre-trained Model
-
-
-### 1. Environment Setup
-
-```bash
-pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
-```
-
-
-### 2. Download the Model
-
-```bash
-git clone https://huggingface.co/jingyaogong/MiniMind2
-```
-
-### 3. Command-line Q&A
-
-```bash
-# load=0: load from pytorch model, load=1: load from transformers-hf model
-python eval_model.py --load 1 --model_mode 2
-```
-
-### 4. Or Start WebUI
-
-```bash
-# You may need `python>=3.10` and install `pip install streamlit`.
-# cd scripts
-streamlit run web_demo.py
-```
-
-## Ⅱ Training from Scratch
-
-### 1. Environment Setup
-
-```bash
-pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
-```
-
-
-Note: Test if Torch can use CUDA
-
-```bash
-import torch
-print(torch.cuda.is_available())
-```
-
-If CUDA is not available, please download the `.whl` file
-from [torch_stable](https://download.pytorch.org/whl/torch_stable.html) and install it. Refer to
-this [link](https://blog.csdn.net/weixin_45456738/article/details/141029610?ops_request_misc=&request_id=&biz_id=102&utm_term=%E5%AE%89%E8%A3%85torch&utm_medium=distribute.pc_search_result.none-task-blog-2~all~sobaiduweb~default-2-141029610.nonecase&spm=1018.2226.3001.4187)
-for guidance.
-
-
-
-### 2. Data Download
-
-Download the required data files from
-the [dataset download link](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files)
-(please `mkdir dataset`) and place them in the `./dataset` directory.
-
-
-Note: Dataset Information
-
-By default, it is recommended to download `pretrain_hq.jsonl` + `sft_mini_512.jsonl` for the fastest Zero-chat model
-reproduction.
-
-You can freely choose data files. Various combinations are provided below, and you can select according to your training
-needs and GPU resources.
-
-
-
-### 3. Start Training
-
-**3.1 Pretraining (Learning Knowledge)**
-
-```bash
-python train_pretrain.py
-```
-
-> Execute pretraining to get `pretrain_*.pth` as the output weights for pretraining (where * represents the model
-> dimension, default is 512).
-
-
-**3.2 Supervised Fine-Tuning (Learning Dialogue Style)**
-
-```bash
-python train_full_sft.py
-```
-
-> Execute supervised fine-tuning to get `full_sft_*.pth` as the output weights for instruction fine-tuning (where `full`
-> represents full parameter fine-tuning).
-
-
-
-Note: Training Information
-
-By default, during training, the model parameters are saved every 100 steps to `./out/***.pth` (each time overwriting
-the old weight file).
-
-For simplicity, only the two training stages are listed here. For other training methods (LoRA, distillation,
-reinforcement learning, fine-tuning inference, etc.), refer to the detailed explanation in the [Experiments] section
-below.
-
-
-
----
-
-### 4. Testing Model Performance
-
-Ensure that the model `*.pth` file you want to test is located in the `./out/` directory.
-Alternatively, you can download and use the `*.pth` files I trained
-from [here](https://www.modelscope.cn/models/gongjy/MiniMind2-PyTorch/files).
-
-```bash
-python eval_model.py --model_mode 1 # Default is 0: Test pretrain model, set to 1: Test full_sft model
-```
-
-
-Note: Testing Information
-
-For more details, you can check the `eval_model.py` script code. The model_mode options are 0: Pretraining model, 1:
-SFT-Chat model, 2: RLHF-Chat model, 3: Reason model.
-
-
-
----
-
-> [!TIP]
-> All training scripts are built using PyTorch's native framework and support multi-GPU acceleration. If your device has
-> N (N>1) GPUs:
-
-Start training with N GPUs on a single machine (DDP, supports multi-node, multi-GPU clusters):
-
-```bash
-torchrun --nproc_per_node N train_xxx.py
-```
-
-
-Note: Others
-
-Start training with N GPUs on a single machine (DeepSpeed):
-
-```bash
-deepspeed --master_port 29500 --num_gpus=N train_xxx.py
-```
-
-Enable wandb to record the training process if needed:
-
-```bash
-# Need to log in: wandb login
-torchrun --nproc_per_node N train_xxx.py --use_wandb
-# and
-python train_xxx.py --use_wandb
-```
-
-By adding the `--use_wandb` parameter, the training process will be recorded, and after training, you can view the
-process on the wandb website. Modify the `wandb_project` and `wandb_run_name` parameters to specify project and run
-names.
-
-
-
-# 📌 Data Overview
-
-## Ⅰ Tokenizer
-
-A tokenizer maps words from natural language into numbers like `0, 1, 36`, which can be understood as the page numbers
-in a "dictionary". You can either construct your own vocabulary to train a tokenizer (code is available
-in `./scripts/train_tokenizer.py`, for educational purposes; MiniMind comes with a built-in tokenizer, so training one
-is unnecessary unless absolutely needed), or you can choose from well-known open-source tokenizers.
-
-The advantage of using a popular dictionary, like the Xinhua or Oxford dictionary, is that the token encoding has good
-compression efficiency, but the downside is that the vocabulary can be very large, with hundreds of thousands of words
-or phrases. On the other hand, a custom tokenizer allows flexibility in controlling the vocabulary's length and content,
-but the trade-off is lower compression efficiency (e.g., "hello" might be split into five independent tokens like "h", "
-e", "l", "l", "o"), and it may miss rare words.
-
-The choice of vocabulary is crucial. The output of an LLM is essentially a multi-class classification problem over the
-vocabulary, with the model decoding the final output into natural language. Since MiniMind's model size needs to be
-strictly controlled, the vocabulary length should be kept short to avoid the embedding layer dominating the model's
-overall parameters. Thus, a smaller vocabulary size is beneficial.
-
-
-Tokenizer Details
-
-Here are the vocabulary sizes of several popular open-source models:
-
-| Tokenizer Model | Vocabulary Size | Source |
-|--------------------|-----------------|-----------------------|
-| yi tokenizer | 64,000 | 01万物 (China) |
-| qwen2 tokenizer | 151,643 | Alibaba Cloud (China) |
-| glm tokenizer | 151,329 | Zhipu AI (China) |
-| mistral tokenizer | 32,000 | Mistral AI (France) |
-| llama3 tokenizer | 128,000 | Meta (USA) |
-| minimind tokenizer | 6,400 | Custom |
-
-> 👉 **2024-09-17 Update**: To avoid ambiguity in previous versions and control model size, all MiniMind models now use
-> the `minimind_tokenizer`. All previous versions using the `mistral_tokenizer` have been deprecated.
-
-```
-# Some personal thoughts
-> Although the `minimind_tokenizer` has a smaller vocabulary size and the encoding/decoding efficiency is weaker than other Chinese-friendly tokenizers like `qwen2` or `glm`, MiniMind has chosen to use this custom tokenizer to maintain a lightweight model overall and avoid an imbalance between the embedding and computation layers.
-> The `minimind_tokenizer` vocabulary size is only 6400, which ensures that the total parameters of the LLM are kept to a minimum (around 25.8M).
-> The training data for this tokenizer (`tokenizer_train.jsonl`) is sourced from the "Jiangshu Large Model Dataset". This part of the data is relatively less important, but you can freely choose any data for training if needed.
-```
-
-
-
-## Ⅱ Pretrain Data
-
-After learning from the poor-quality pretraining data of MiniMind-V1, which resulted in nonsensical outputs, I decided
-not to use large-scale unsupervised datasets for pretraining post-`2025-02-05`. Instead, I extracted the Chinese portion
-of the [Jiangshu Large Model Dataset](https://www.modelscope.cn/datasets/deepctrl/deepctrl-sft-data), cleaned the
-content to include only characters of length `<512`, resulting in around 1.6GB of high-quality pretraining data, saved
-as `pretrain_hq.jsonl`.
-
-The data format for `pretrain_hq.jsonl` is:
-
-```bash
-{"text": "如何才能摆脱拖延症？ 治愈拖延症并不容易，但以下建议可能有所帮助..."}
-```
-
-## Ⅲ SFT Data
-
-The [Jiangshu Large Model SFT Dataset](https://www.modelscope.cn/datasets/deepctrl/deepctrl-sft-data) is a complete,
-well-formatted dataset for large model training and research. It includes approximately 10M Chinese sentences and 2M
-English sentences. However, the provided format is messy, and using the entire dataset for SFT would be too costly.
-
-I have cleaned this dataset, removing noisy entries with special characters and symbols, and only kept content with a
-length `<512`. This cleaned dataset is exported as `sft_512.jsonl` (~7.5GB).
-
-Additionally, I have collected around 1M high-quality dialogue data from Qwen2/2.5, cleaned and exported the content
-with lengths `<2048` into `sft_2048.jsonl` (~9GB) and those with lengths `<1024` into `sft_1024.jsonl` (~5.5GB).
-
-Further cleaning of these SFT datasets (only keeping content with a higher ratio of Chinese characters) resulted
-in `sft_mini_512.jsonl` (~1.2GB).
-
-The data format for all SFT files `sft_X.jsonl` is as follows:
-
-```text
-{
- "conversations": [
- {"role": "user", "content": "你好"},
- {"role": "assistant", "content": "你好！"},
- {"role": "user", "content": "再见"},
- {"role": "assistant", "content": "再见！"}
- ]
-}
-```
-
-## Ⅳ RLHF Data
-
-The [Magpie-DPO Dataset](https://www.modelscope.cn/datasets/Magpie-Align/MagpieLM-DPO-Data-v0.1) contains around 200k
-preference data generated from Llama3.1-70B/8B and can be used for training reward models to optimize response quality
-according to human preferences.
-
-I have cleaned this dataset by combining data with a total length `<3000` into `dpo.jsonl` (~0.9GB), which
-contains `chosen` (preferred) and `rejected` (rejected) replies.
-
-The data format for `dpo.jsonl` is:
-
-```text
-{
- "chosen": [
- {"content": "Q", "role": "user"},
- {"content": "good answer", "role": "assistant"}
- ],
- "rejected": [
- {"content": "Q", "role": "user"},
- {"content": "bad answer", "role": "assistant"}
- ]
-}
-```
-
-## Ⅴ Reasoning Dataset
-
-The excitement over **DeepSeek** in February 2025 has greatly influenced my interest in RL-guided reasoning models. I
-have already replicated **R1-Zero** using Qwen2.5. If time allows and if it works, I plan to update MiniMind with a
-reasoning model trained with RL, rather than a distilled model.
-
-Currently, the quickest and cost-effective approach is still distillation (black-box style). But due to the popularity
-of **R1**, I’ve combined several distilled datasets related to **R1**, including:
-
-- [R1-Llama-70B](https://www.modelscope.cn/datasets/Magpie-Align/Magpie-Reasoning-V2-250K-CoT-Deepseek-R1-Llama-70B)
-- [R1-Distill-SFT](https://www.modelscope.cn/datasets/AI-ModelScope/R1-Distill-SFT)
-- [Alpaca-Distill-R1](https://huggingface.co/datasets/shareAI/Alpaca-Distill-R1-ZH)
-- [deepseek_r1_zh](https://huggingface.co/datasets/jinliuxi/deepseek_r1_zh)
-
-After combining these, I exported the file as `r1_mix_1024.jsonl`. The format of this file is the same as `sft_X.jsonl`.
-
-## Ⅵ Additional Datasets
-
-For more datasets related to Chinese LLMs, you can refer
-to [HqWu-HITCS/Awesome-Chinese-LLM](https://github.com/HqWu-HITCS/Awesome-Chinese-LLM), which collects and organizes
-open-source models, applications, datasets, and tutorials for Chinese LLMs. It's comprehensive and regularly updated.
-Big respect!
-
----
-
-## Ⅷ Dataset Download
-
-> [!NOTE]
-> After `2025-02-05`, MiniMind’s open-source datasets for final training are provided, so there is no need for
-> you to preprocess large datasets by yourself anymore. This helps avoid redundant work.
-
-MiniMind Training Datasets are available for download from:
-
-Dataset ([ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind_dataset/tree/main))
-
-> You don’t need to clone everything, just download the necessary files.
-
-Place the downloaded dataset files in the `./dataset/` directory (✨ required files are marked):
-
-```bash
-./dataset/
-├── dpo.jsonl (909MB)
-├── lora_identity.jsonl (22.8KB)
-├── lora_medical.jsonl (34MB)
-├── pretrain_hq.jsonl (1.6GB, ✨)
-├── r1_mix_1024.jsonl (340MB)
-├── sft_1024.jsonl (5.6GB)
-├── sft_2048.jsonl (9GB)
-├── sft_512.jsonl (7.5GB)
-├── sft_mini_512.jsonl (1.2GB, ✨)
-└── tokenizer_train.jsonl (1GB)
-```
-
-
- Dataset Descriptions
-
-* `dpo.jsonl` -- RLHF dataset
-* `lora_identity.jsonl` -- Self-identity dataset (e.g., "Who are you? I'm MiniMind..."), recommended for LoRA training (
- also applicable for full parameter SFT)
-* `lora_medical.jsonl` -- Medical Q&A dataset, recommended for LoRA training (also applicable for full parameter SFT)
-* `pretrain_hq.jsonl`✨ -- Pretraining dataset from Jiangshu Technology
-* `r1_mix_1024.jsonl` -- DeepSeek-R1-1.5B distilled dataset (max length 1024 characters)
-* `sft_1024.jsonl` -- Qwen2.5 distilled data (subset of sft_2048, max length 1024)
-* `sft_2048.jsonl` -- Qwen2.5 distilled data (max length 2048)
-* `sft_512.jsonl` -- Jiangshu SFT dataset (max length 512)
-* `sft_mini_512.jsonl`✨ -- Minimal Jiangshu + Qwen2.5 distilled dataset (max length 512)
-* `tokenizer_train.jsonl` -- From Jiangshu Large Model Dataset (not recommended for custom tokenizer training)
-
-
-
-
-
-
-
-Explanation & Recommended Training Plans
-
-* The MiniMind2 Series has been trained on approximately 20GB of corpus, or about 4B tokens, corresponding to the data
- combination results above (Cost: 💰💰💰💰💰💰💰💰, Effect: 😊😊😊😊😊😊).
-
-* For the fastest Zero-model implementation from scratch, it is recommended to use the data combination
- of `pretrain_hq.jsonl` + `sft_mini_512.jsonl`. The specific costs and effects can be seen in the table below (Cost:
- 💰, Effect: 😊😊).
-
-* For those with sufficient computational resources or more focus on results, it is advisable to fully reproduce
- MiniMind2 with the first option; if you only have a single GPU or prefer a quick reproduction within a short time, the
- second option is strongly recommended.
-
-* [Compromise Plan] You can also freely combine medium-sized data like `sft_mini_512.jsonl`, `sft_1024.jsonl` for
- training (Cost: 💰💰💰, Effect: 😊😊😊😊).
-
-
-
-# 📌 Model Structure
-
-MiniMind-Dense (like [Llama3.1](https://ai.meta.com/blog/meta-llama-3-1/)) uses the Transformer Decoder-Only structure,
-which differs from GPT-3 in the following aspects:
-
-* It adopts GPT-3's pre-normalization method, meaning normalization is done on the input of each Transformer sub-layer
- instead of on the output. Specifically, RMSNorm normalization function is used.
-* The SwiGLU activation function is used instead of ReLU to improve performance.
-* Like GPT-Neo, absolute position embeddings are removed and replaced with rotary position embeddings (RoPE), which
- perform better when handling inference beyond the training length.
-
----
-
-The MiniMind-MoE model is based on the MixFFN mixture of experts module from Llama3
-and [Deepseek-V2/3](https://arxiv.org/pdf/2405.04434).
-
-* DeepSeek-V2, in terms of feedforward networks (FFN), adopts finer-grained expert splitting and shared expert isolation
- techniques to improve the performance of Experts.
-
----
-
-The overall structure of MiniMind remains consistent, with only minor adjustments made to RoPE computation, inference
-functions, and FFN layers.
-The structure is as shown in the figure below (redrawn):
-
-
-
-
-For model configuration modifications, see [./model/LMConfig.py](./model/LMConfig.py).
-Reference model parameter versions are shown in the table below:
-
-| Model Name | params | len_vocab | n_layers | d_model | kv_heads | q_heads | share+route |
-|-------------------|--------|-----------|----------|---------|----------|---------|-------------|
-| MiniMind2-Small | 26M | 6400 | 8 | 512 | 2 | 8 | - |
-| MiniMind2-MoE | 145M | 6400 | 8 | 640 | 2 | 8 | 1+4 |
-| MiniMind2 | 104M | 6400 | 16 | 768 | 2 | 8 | - |
-| minimind-v1-small | 26M | 6400 | 8 | 512 | 8 | 16 | - |
-| minimind-v1-moe | 4×26M | 6400 | 8 | 512 | 8 | 16 | 1+4 |
-| minimind-v1 | 108M | 6400 | 16 | 768 | 8 | 16 | - |
-
-# 📌 Experiment
-
-## Ⅰ Training Cost
-
-- **Time Unit**: Hours (h).
-- **Cost Unit**: RMB (￥); 7￥ ≈ 1 USD.
-- **3090 Rental Unit Price**: ≈ 1.3￥/h (subject to real-time market rates).
-- **Reference Standard**: The table only shows the actual training time for the `pretrain` and `sft_mini_512` datasets.
- Other times are estimated based on dataset size (there may be some discrepancies).
-
-> Based on 3090 (single card) cost calculation
-
-| Model Name | params | pretrain | sft_mini_512 | sft_512 | sft_1024 | sft_2048 | RLHF |
-|-----------------|--------|------------------|------------------|---------------|-------------------|------------------|---------------|
-| MiniMind2-Small | 26M | ≈1.1h ≈1.43￥ | ≈1h ≈1.3￥ | ≈6h ≈7.8￥ | ≈4.58h ≈5.95￥ | ≈7.5h ≈9.75￥ | ≈1h ≈1.3￥ |
-| MiniMind2 | 104M | ≈3.9h ≈5.07￥ | ≈3.3h ≈4.29￥ | ≈20h ≈26￥ | ≈15h ≈19.5￥ | ≈25h ≈32.5￥ | ≈3h ≈3.9￥ |
-
----
-
-
-Training Cost Summary & Prediction
-
-> MiniMind2-Small Parameters
->> `pretrain_hq` + `sft_mini_512` dataset
- Single 3090 GPU (1 epoch) + 2.1 hours + Cost: 2.73 RMB
- You can train the MiniMind-Zero-0.025B model from scratch!!!
-
-> MiniMind2-Small Parameters
->> `pretrain_hq` + `sft_512` + `sft_2048` + `dpo` dataset
- Single 3090 GPU (2 epochs) + approximately 38.16 hours + Cost: 49.61 RMB
- You can train the MiniMind2-Small-0.025B model from scratch!!!
-
-> MiniMind2 Parameters
->> `pretrain_hq` + `sft_512` + `sft_2048` + `dpo` dataset
- Single 3090 GPU (2 epochs) + approximately 122 hours + Cost: 158.6 RMB
- You can train the MiniMind2-0.1B model from scratch!!!
-
-
-
-
-✨With a single NVIDIA 3090 GPU, you can train `MiniMind-Zero` from scratch in just `2 hours` and for a cost of
-only `3 RMB`, achieving ChatBot functionality!
-
-✨PS: If training on 8 GPUs with 4090s, the total time can be compressed to under 10 minutes! (Despite the shorter time,
-the cost is still around 3 RMB, which is comparable to the single GPU cost.)
-
-✨This enables ultra-low barriers to entry, making it possible for everyone to experiment with large models! This is the
-original purpose behind the creation of the MiniMind series!
-
-✨The `MiniMind-Zero` model, which costs only `3 RMB`, is not a gimmick! Chat test results:
-
-```textmate
-👶: Please introduce yourself.
-🤖️: As an artificial intelligence, I don't have a real life or self-awareness, so I don't have my own life. I am designed to help users answer questions, provide information, engage in conversations, etc. My design and functionality are built using computer science, machine learning, artificial intelligence technologies, and algorithms, which I implement through programming and algorithms.
-
-👶: Recommend some local specialties from Hangzhou.
-🤖️: Hangzhou has many delicious local specialties, such as chicken skewers, eel heads, water-boiled fish, tofu pudding, etc. These foods are all very tasty, with unique flavors and textures, definitely worth trying.
-
-👶: Please explain the concept of "large language models".
-🤖️: A large language model is an AI technology that uses large amounts of text data to train the model, and then continuously optimizes the model through iterations. Its goal is to generate text based on the input data to meet specific needs, and the text can be about any topic you are interested in.
-
-👶: Introduce the history of the United States, please.
-🤖️: You mentioned "Introok's the believeations of theument." This name originates from the ancient Chinese "groty of of the change."
-```
-
-Fast and effective, it is still possible to further compress the training process by obtaining smaller and
-higher-quality datasets.
-The Zero model weights are saved as `full_sft_512_zero.pth` (see the MiniMind model file link below). Feel free to
-download and test the model's performance.
-
-## Ⅱ Main Training Steps
-
-### **1. Pretraining**:
-
-The first task for LLM is not to interact directly with humans, but to fill the network parameters with knowledge. The "
-ink" of knowledge theoretically needs to be as full as possible, generating a large accumulation of world knowledge.
-Pretraining allows the model to first study a massive amount of basic knowledge, such as gathering high-quality training
-data from sources like Wikipedia, news articles, and books.
-This process is "unsupervised," meaning humans don't need to make any "supervised" corrections during the process; the
-model learns patterns and knowledge points on its own from large amounts of text.
-The goal at this stage is simple: **learn word chaining**. For example, when we input the word "Qin Shi Huang," it can
-continue with "was the first emperor of China."
-
-```bash
-torchrun --nproc_per_node 1 train_pretrain.py # 1 represents single-card training, adjust according to hardware (set >=2)
-# or
-python train_pretrain.py
-```
-
-> The trained model weights are saved every `100 steps` by default as: `pretrain_*.pth` (the * represents the specific
-> model dimension, and each new save will overwrite the previous one).
-
-### **2. Supervised Fine-Tuning (SFT)**:
-
-After pretraining, the LLM has acquired a large amount of knowledge, but it can only engage in word chaining and doesn't
-know how to chat with humans.
-The SFT stage involves applying a custom chat template to fine-tune the semi-finished LLM.
-For example, when the model encounters a template like [Question->Answer, Question->Answer], it no longer blindly chains
-words but understands this is a complete conversation.
-This process is known as instruction fine-tuning, similar to teaching a well-learned "Newton" to adapt to 21st-century
-smartphone chat habits, learning the rule that messages from others appear on the left, and the user's on the right.
-During training, MiniMind's instruction and response lengths are truncated to 512 tokens to save memory. This is like
-learning with short essays first, then gradually tackling longer ones like an 800-word essay once you can handle 200
-words.
-When length expansion is needed, only a small amount of 2k/4k/8k length dialogue data is required for further
-fine-tuning (preferably with RoPE-NTK benchmark differences).
-> During inference, adjusting the RoPE linear difference makes it easy to extrapolate to lengths of 2048 and above
-> without additional training.
-
-```bash
-torchrun --nproc_per_node 1 train_full_sft.py
-# or
-python train_full_sft.py
-```
-
-> The trained model weights are saved every `100 steps` by default as: `full_sft_*.pth` (the * represents the specific
-> model dimension, and each new save will overwrite the previous one).
-
-## Ⅲ Other Training Steps
-
-### **3. Reinforcement Learning from Human Feedback (RLHF)**
-
-In the previous training steps, the model has acquired basic conversational abilities, but these are entirely based on
-word chaining, without positive or negative reinforcement examples.
-At this point, the model doesn't know what answers are good or bad. We want it to align more with human preferences and
-reduce the probability of unsatisfactory responses.
-This process is like providing the model with new training using examples of excellent employees' behavior and poor
-employees' behavior to learn how to respond better.
-Here, we use RLHF’s Direct Preference Optimization (DPO).
-Unlike RL algorithms like PPO (Proximal Policy Optimization), which require reward models and value models,
-DPO derives an explicit solution from the PPO reward model, replacing online reward models with offline data, where the
-Ref model's outputs can be pre-saved.
-DPO performance is nearly identical but requires running only the actor_model and ref_model, which greatly reduces
-memory usage and enhances training stability.
-> **Note**: The RLHF step **is not required**, as it typically does not improve the model’s "intelligence" but is used
-> to improve the model's "politeness," which can have pros (alignment with preferences, reducing harmful content) and
-> cons (expensive sample collection, feedback bias, loss of diversity).
-
-```bash
-torchrun --nproc_per_node 1 train_dpo.py
-# or
-python train_dpo.py
-```
-
-> The trained model weights are saved every `100 steps` by default as: `rlhf_*.pth` (the * represents the specific model
-> dimension, and each new save will overwrite the previous one).
-
-### **4. Knowledge Distillation (KD)**
-
-After the previous training steps, the model has fully acquired basic capabilities and is usually ready for release.
-Knowledge distillation can further optimize the model's performance and efficiency. Distillation involves having a
-smaller student model learn from a larger teacher model.
-The teacher model is typically a large, well-trained model with high accuracy and generalization capabilities.
-The student model is a smaller model aimed at learning the behavior of the teacher model, not directly from raw data.
-In SFT learning, the model’s goal is to fit the hard labels (e.g., real category labels like 0 or 6400) in word token
-classification.
-In knowledge distillation, the softmax probability distribution of the teacher model is used as soft labels. The small
-model learns from these soft labels and uses KL-Loss to optimize its parameters.
-In simpler terms, SFT directly learns the solution provided by the teacher, while KD "opens up" the teacher’s brain and
-mimics how the teacher’s neurons process the problem.
-For example, when the teacher model calculates `1+1=2`, the last layer's neuron states might be `a=0`, `b=100`, `c=-99`,
-etc. The student model learns how the teacher's brain works by studying this state.
-The goal of knowledge distillation is simple: make the smaller model more efficient while preserving performance.
-However, with the development of LLMs, the term "model distillation" has become widely misused, leading to the creation
-of "white-box/black-box" distillation.
-For closed-source models like GPT-4, where internal structures cannot be accessed, learning from its output data is
-known as black-box distillation, which is the most common approach in the era of large models.
-Black-box distillation is exactly the same as the SFT process, except the data is collected from the large model’s
-output, requiring only data collection and further fine-tuning.
-Note to change the base model loaded to `full_sft_*.pth`, as distillation is performed based on the fine-tuned model.
-The `./dataset/sft_1024.jsonl` and `./dataset/sft_2048.jsonl` datasets, collected from the qwen2.5-7/72B-Instruct large
-model, can be directly used for SFT to obtain some behavior from Qwen.
-
-```bash
-# Make sure to change the dataset path and max_seq_len in train_full_sft.py
-torchrun --nproc_per_node 1 train_full_sft.py
-# or
-python train_full_sft.py
-```
-
-> The trained model weights are saved every `100 steps` by default as: `full_sft_*.pth` (the * represents the specific
-> model dimension, and each new save will overwrite the previous one).
-
-This section emphasizes MiniMind’s white-box distillation code `train_distillation.py`. Since MiniMind doesn’t have a
-powerful teacher model within the same series, the white-box distillation code serves as a learning reference.
-
-```bash
-torchrun --nproc_per_node 1 train_distillation.py
-# or
-python train_distillation.py
-```
-
-### **5. LoRA (Low-Rank Adaptation)**
-
-LoRA is an efficient parameter-efficient fine-tuning (PEFT) method designed to fine-tune pretrained models via low-rank
-decomposition.
-Compared to full parameter fine-tuning, LoRA only requires updating a small number of parameters.
-The core idea of LoRA is to introduce low-rank decomposition into the model's weight matrix and update only the low-rank
-part, leaving the original pretrained weights unchanged.
-The code can be found in `./model/model_lora.py` and `train_lora.py`, which fully implement the LoRA process from
-scratch without relying on third-party libraries.
-
-```bash
-torchrun --nproc_per_node 1 train_lora.py
-# or
-python train_lora.py
-```
-
-> The trained model weights are saved every `100 steps` by default as: `lora_xxx_*.pth` (the * represents the specific
-> model dimension, and each new save will overwrite the previous one).
-
-Many people are puzzled: how can a model learn private domain knowledge? How should datasets be prepared? How to
-transfer general models into specialized domain models?
-Here are some examples. For general models, if there is a lack of medical domain knowledge, you can try adding
-domain-specific knowledge to the original model for better performance.
-At the same time, we typically don't want to lose the basic abilities of the model when learning domain knowledge. LoRA
-can address this issue well.
-Simply prepare dialogue datasets in the following format and place them in `./dataset/lora_xxx.jsonl`, then
-start `python train_lora.py`.
-Training will result in new model weights in `./out/lora/lora_xxx.pth`.
-
-**Medical Scenario**
-
-```
- {"conversations": [{"role": "user", "content": "What is the ideal pillow height for cervical spondylosis?"}, {"role": "assistant", "content": "The pillow height for cervical spondylosis should be..."}]}
- {"conversations
-
-": [{"role": "user", "content": "What is xxx?"}, {"role": "assistant", "content": "xxx..."}]}
-```
-
-**Self-awareness Scenario**
-
-```
- {"conversations": [{"role": "user", "content": "What is your name?"}, {"role": "assistant", "content": "I am MiniMind..."}]}
- {"conversations": [{"role": "user", "content": "Who are you?"}, {"role": "assistant", "content": "I am..."}]}
-```
-
-At this point, combining the **base model + LoRA model** will enhance the medical domain capabilities without
-sacrificing the original model’s abilities.
-You can evaluate the model’s performance using `eval_model.py`.
-
-```bash
-# Ensure model_mode matches the model used during train_lora to avoid mismatch
-python eval_model.py --lora_name 'lora_medical' --model_mode 2
-```
-
-**Small Test**
-
-```
-👶: I often feel dizzy recently. What might be the cause?
-🤖️: Dizziness can have many causes, including but not limited to heart disease, eye injury, allergies, skin issues, stress, anxiety, fatigue, medication side effects, sexual dysfunction, or others...
-
----
-
-👶: What is your relationship with OpenAI?
-🤖️: I am MiniMind, an AI assistant developed by Jingyao Gong. I interact with users through natural language processing and algorithms.
-```
-
-PS: If needed, you can also fine-tune the full parameters using `full_sft` (but mix general knowledge to prevent
-overfitting on domain-specific data, which could reduce the generality of the model).
-
-### **6. Training the Reasoning Model (Reasoning Model)**
-
-DeepSeek-R1 is so popular that it has almost redefined the future paradigm for LLMs.
-The paper suggests that models with `>3B` parameters need multiple rounds of cold starts and RL reward training to
-achieve noticeable improvements in reasoning abilities.
-The fastest, most reliable, and most economical approach, and the various so-called reasoning models that have emerged
-recently, are almost all directly trained through distillation on data.
-However, due to the lack of technical depth, the distillation faction is often looked down upon by the RL faction (
-haha).
-I quickly attempted this on the Qwen series 1.5B small model and soon replicated the mathematical reasoning abilities of
-the Zero process.
-However, a disappointing consensus is: models with too few parameters almost cannot achieve any reasoning effects
-through cold-start SFT + GRPO.
-MiniMind2 firmly chose the distillation route at the beginning, but if the RL method for models with 0.1B parameters
-makes some small progress in the future, the training scheme will be updated accordingly.
-
-To do distillation, you need to prepare data in the same format as the SFT phase, as described earlier. The data format
-is as follows:
-
-```json lines
-{
- "conversations": [
- {
- "role": "user",
- "content": "Hello, I'm Xiaofang, nice to meet you."
- },
- {
- "role": "assistant",
- "content": "\nHello! I am MiniMind-R1-Lite-Preview, an intelligent assistant independently developed by a Chinese individual. I'm happy to provide services for you!\n\n\nHello! I am MiniMind-R1-Lite-Preview, an intelligent assistant independently developed by a Chinese individual. I'm happy to provide services for you!\n"
- }
- ]
-}
-```
-
-The reply template for the reasoning model R1 is:
-
-```text
-\nThinking process\n\n
-\nFinal answer\n
-```
-
-In GRPO, this is done by setting up a reward function that ensures the model adheres to the thinking and answering
-tags (the reward values should be higher in the early cold-start stages).
-
-Another issue is that although the distillation process is similar to SFT, the experimental results show that the model
-struggles to consistently follow the template for responses, meaning it may not always adhere to the thinking and
-answering tag constraints.
-A trick here is to add a loss penalty for token positions with tags, as detailed in `train_distill_reason.py`:
-
-```text
-# Add additional penalty at positions corresponding to sp_ids
-...
-loss_mask[sp_ids] = 10 # Penalty coefficient
-```
-
-Another tip is that, since the reasoning data only filters for data of length `<1024`, there is less multi-turn dialogue
-and English data.
-Therefore, `r1_mix_1024.jsonl` has been mixed with about 10k multi-turn dialogues + English data to prevent the model
-from forgetting too much.
-
-The script is set by default to distill reasoning abilities from the RLHF-based pre-trained model. To start training,
-just run:
-
-```bash
-torchrun --nproc_per_node 1 train_distill_reason.py
-# or
-python train_distill_reason.py
-```
-
-> The trained model weights are saved every `100 steps` by default as: `reason_*.pth` (* being the specific dimension of
-> the model; each time a new file is saved, it will overwrite the old one).
-
-Test it:
-
-```text
-👶: Please introduce yourself.
-🤖️:
-Hmm, the user asks me to introduce myself. I need to explain myself. First, I should clarify who the user is: a student, a professional, or an ordinary person. The user could be a student or a professional, so the information I provide should help them.
-Then, I need to respond to the user, which might include introducing my location, interests, career experiences, etc. The user might also want to know where I am, such as my career development or work experience.
-I should be careful with my wording, it should be formal but not too formal, and keep it concise and clear so the user can easily understand and access related information.
-Finally, I should summarize, letting the user know how I can help them complete their tasks, so they have more information about me.
-
-
-I am **"Your Name"**, what is your name?
-
-
-👶: Which subject are you better at?
-🤖️:
-Hello! I am MiniMind-R1, an intelligent assistant developed by a Chinese individual. If you have any questions, I will do my best to assist you.
-
-
-Hello! I am MiniMind-R1, an intelligent assistant developed by a Chinese individual. If you have any questions, I will do my best to assist you.
-
-```
-
----
-
-## Ⅳ Model Parameter Settings
-
-📋 Regarding LLM parameter configurations, an interesting paper, [MobileLLM](https://arxiv.org/pdf/2402.14905), provides
-a detailed study and experiment.
-The Scaling Law has unique patterns for small models.
-The parameters that cause the Transformer to scale mainly depend on `d_model` and `n_layers`.
-
-* `d_model`↑ + `n_layers`↓ -> Short and fat
-* `d_model`↓ + `n_layers`↑ -> Tall and thin
-
-The Scaling Law paper from 2020 suggests that the training data volume, parameter size, and number of training
-iterations are the key factors determining performance, with the model architecture having almost negligible impact.
-However, this law doesn't seem to fully apply to small models.
-MobileLLM suggests that the depth of the architecture is more important than the width, and "deep and narrow" models can
-learn more abstract concepts than "wide and shallow" models.
-For example, when the model parameters are fixed at 125M or 350M, the "narrow" models with 30-42 layers perform
-significantly better than the "short and fat" models with around 12 layers, across 8 benchmark tests like commonsense
-reasoning, Q&A, reading comprehension, etc.
-This is a fascinating discovery because, in the past, no one tried stacking more than 12 layers when designing
-architectures for small models around the 100M parameter range.
-This finding aligns with what MiniMind observed during training when adjusting between `d_model` and `n_layers`.
-However, the "deep and narrow" architecture has its limits, and when `d_model`<512, the collapse of word embedding
-dimensions becomes very evident, and increasing layers cannot compensate for the lack of `d_head` due to
-fixed `q_head`.
-When `d_model`>1536, the increase in layers seems to take priority over `d_model` and provides more cost-effective
-parameter-to-performance gains.
-
-* Therefore, MiniMind sets small models with `dim=512`, `n_layers=8` to strike a balance between "small size" and "
- better performance."
-* Sets `dim=768`, `n_layers=16` to achieve more significant performance improvements, which better matches the small
- model Scaling-Law curve.
-
-For reference, the parameter settings for GPT-3 are shown in the table below:
-
-
----
-
-## Ⅴ Training Results
-
-> `MiniMind2` model training loss trends (the loss is for reference only as the dataset was updated and cleaned several
-> times after training).
-
-| Models | Pretrain (length-512) | SFT (length-512) |
-|-----------------|----------------------------------------------------|----------------------------------------------------|
-| MiniMind2-Small | | |
-| MiniMind2 | | |
-
-### Training Completed - Model Collection
-
-> Considering that many people have reported slow speeds with Baidu Cloud, all MiniMind2 models and beyond will be
-> hosted on ModelScope/HuggingFace.
-
----
-
-#### ① Native PyTorch Models
-
-MiniMind2 model
-weights ([ModelScope](https://www.modelscope.cn/models/gongjy/MiniMind2-PyTorch) | [HuggingFace](https://huggingface.co/jingyaogong/MiniMind2-Pytorch))
-
-MiniMind-V1 model weights ([Baidu Pan](https://pan.baidu.com/s/1KUfSzEkSXYbCCBj0Pw-9fA?pwd=6666))
-
-
-Torch File Naming Reference
-
-| Model Name | params | pretrain_model | sft_model | rl_model | reason_model | lora_model |
-|-----------------|--------|------------------------|------------------------|--------------------|------------------|--------------------|
-| MiniMind2-small | 26M | `pretrain_512.pth` | `full_sft_512.pth` | `rlhf_512.pth` | `reason_512.pth` | `lora_xxx_512.pth` |
-| MiniMind2-MoE | 145M | `pretrain_640_moe.pth` | `full_sft_640_moe.pth` | `rlhf_640_moe.pth` | - | - |
-| MiniMind2 | 104M | `pretrain_768.pth` | `full_sft_768.pth` | `rlhf_768.pth` | `reason_768.pth` | `lora_xxx_768.pth` |
-
-| Model Name | params | pretrain_model | Single-turn Dialogue SFT | Multi-turn Dialogue SFT | rl_model |
-|-------------------|--------|------------------------|------------------------------------|-----------------------------------|--------------|
-| minimind-v1-small | 26M | `pretrain_512.pth` | `single_chat/full_sft_512.pth` | `multi_chat/full_sft_512.pth` | `rl_512.pth` |
-| minimind-v1-moe | 4×26M | `pretrain_512_moe.pth` | `single_chat/full_sft_512_moe.pth` | `multi_chat/full_sft_512_moe.pth` | - |
-| minimind-v1 | 108M | `pretrain_768.pth` | `single_chat/full_sft_768.pth` | `multi_chat/full_sft_768.pth` | `rl_768.pth` |
-
-
-
-#### ② Transformers Models
-
-MiniMind
-Series ([ModelScope](https://www.modelscope.cn/collections/MiniMind-b72f4cfeb74b47) | [HuggingFace](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5))
-
-
----
-
-# 📌 Eval
-
-## Ⅰ RLHF vs SFT Comparison
-
-The test is based on comparing `MiniMind2 series` models with "full_sft" and "rlhf". The random seed is fixed.
-
-
-Q&A Test
-
-[A0] MiniMind2：full_sft_640
-
-[A1] MiniMind2：rlhf_640
-
- ```text
- [Q]: What is your name?
- [A0]: I am a large language model capable of answering various questions, providing information and assistance. I can answer a wide range of questions, provide information, help you solve problems, and more.
- [A1]: I am a text-based AI model, and I don't have personal experiences or emotions, so I cannot feel or experience emotions. My design is to assist users by solving problems, providing information, and engaging in conversations. If you have any questions or need help, feel free to let me know!
-
- [Q]: Which subject are you better at?
- [A0]: As an AI model, I do not have a specific "specialty". I am designed to provide information, answer questions, engage in conversation, perform tasks, and interact with users. However, I can offer assistance in various subjects including providing information, solving problems, engaging in discussions, giving advice, learning new knowledge, writing code, and more. If you have a specific subject or topic in mind, feel free to provide more details.
- [A1]: As an AI, I do not have personal capabilities or interests, but I can provide knowledge on subjects such as mathematics, science, computer science, linguistics, psychology, and more. If you're interested in mathematics, I can offer some basic knowledge like algebra, geometry, probability, statistics, etc. If you need more in-depth learning, I can also provide relevant online courses, books, research papers, and other resources. Let me know what specific help you need, and I will do my best to assist you.
-
- [Q]: How does Lu Xun's "A Madman's Diary" criticize feudal ethics?
- [A0]: Lu Xun's "A Madman's Diary" is a work that advocates for human values and moral responsibility. In this work, Lu Xun explores profound themes such as human nature, morality, justice, sacrifice, and responsibility through detailed writing. Here are several ways in which Lu Xun critiques feudal ethics in "A Madman's Diary":
- 1. **Social Reality and Moral Concepts**: Lu Xun addresses social inequality, power dynamics, and human nature through his narrative. He argues that social inequality is not only unjust but also a form of social injustice affecting individuals. He calls for collective effort to build a harmonious society through education, publicity, and laws.
- 2. **Morality and Responsibility**: Through his narrative, Lu Xun highlights the importance of moral education and emphasizes personal responsibility.
- 3. **The Individual and Society**: Lu Xun delves into the complex relationship between the individual and society, advocating for collective efforts to contribute to social harmony.
- Lu Xun’s "A Madman's Diary" not only achieved great literary success but also inspired later literary creations with its profound impact on thoughts and societal reflections.
- [A1]: Lu Xun's "A Madman's Diary" is a work that critiques the feudal society's various issues and conflicts, reflecting social inequality and human nature. In this work, Lu Xun critiques feudal society’s problems, like "Frederick Johnson’s 'Eagle Dillon', 'Nazareth', 'Beckham', 'George Wilson'", emphasizing the inequality and complexity of human nature within feudal society. The work criticizes feudal society's corruption, human nature, and the inherent contradictions of the era. This critique reflects Lu Xun’s respect for social issues and his deep understanding of human nature. "A Madman's Diary" is a critical work that raises awareness and challenges the established norms of feudalism and has greatly influenced future critiques.
-
- [Q]: I’ve had a cough for two weeks, should I go to the hospital for a checkup?
- [A0]: A prolonged cough does require some checkups, and if your cough persists or is accompanied by other symptoms such as difficulty breathing, it's recommended to see a doctor. Based on your symptoms, physical examination, and possible imaging tests (e.g., X-rays, MRIs, or CT scans), the doctor will make a diagnosis. In the meantime, you can try the following to alleviate symptoms:
- 1. **Rest**: Ensure you get enough sleep.
- 2. **Stay hydrated**: Dehydration can worsen coughing.
- 3. **Warm compress**: Apply a warm compress to the sore area to relieve throat pain.
- 4. **Avoid irritants**: Avoid substances that may irritate the throat, such as certain chemicals.
- 5. **Maintain humidity**: Keep the room well-ventilated, and use a humidifier if necessary.
- If symptoms persist or worsen, seek medical attention immediately. Wishing you a speedy recovery!
- [A1]: If your cough has lasted for two weeks, it's usually recommended to see a doctor, as it may affect your breathing and heart rate. You can take the following steps to manage your cough:
- 1. **Rest**: Gently press on the coughing area with a tissue or elbow.
- 2. **Use medication**: If possible, use medications like antihistamines to relieve the cough (under a doctor's guidance).
- 3. **Warm compress**: Use a warm compress to alleviate pain and swelling.
- 4. **Medication treatment**: If the cough persists, use anti-inflammatory or antihistamine medications under the guidance of a doctor.
- 5. **Maintain a healthy lifestyle**: Get sufficient sleep, eat a balanced diet, and avoid smoking and excessive alcohol consumption to help reduce the chances of coughing.
- Please remember these suggestions are not substitutes for professional medical advice. If your symptoms persist or worsen, it is strongly advised to see a doctor.
- ```
-
-
-
-👉 Summary of Results
-
-The full_sft model performs better in terms of simplicity and information accuracy; the rlhf model tends to provide more
-background information in its responses, but its accuracy needs improvement.
-Overall, after RLHF, the model tends to learn to say more polite but irrelevant things to please the "conversation",
-while sacrificing some information accuracy.
-There is no free lunch, and further improvement in the quality of the RLHF dataset is necessary. It's also important to
-accept that some loss in model capability is inevitable (to varying degrees).
-The difference between DPO and online PPO lies in the fact that both the reject and chosen are prepared offline, which
-causes a significant distribution difference compared to the minimind model's original output.
-Simply put, the DPO algorithm trains the model using RL by watching "recorded" games of a table tennis world champion,
-rather than using the PPO method where the reward model acts as a "coach" to correct the model's moves during RL.
-
-## Ⅱ Subjective Sample Evaluation
-
-🏃The following tests were completed on February 9, 2025. New models released after this date will not be included in the
-tests unless there is a special need.
-
-[A] [MiniMind2 (0.1B)](https://www.modelscope.cn/models/gongjy/MiniMind2-PyTorch)
-[B] [MiniMind2-MoE (0.15B)](https://www.modelscope.cn/models/gongjy/MiniMind2-PyTorch)
-[C] [MiniMind2-Small (0.02B)](https://www.modelscope.cn/models/gongjy/MiniMind2-PyTorch)
-[D] [minimind-v1-small(0.02B)](https://pan.baidu.com/s/1_COe0FQRDmeapSsvArahCA?pwd=6666)
-[E] [minimind-v1-moe(0.1B)](https://pan.baidu.com/s/1tqB-GMvuiGQBvEl-yZ-oBw?pwd=6666)
-[F] [minimind-v1(0.1B)](https://pan.baidu.com/s/1p713loS7EfwHQf3G9eYI3Q?pwd=6666)
-[G] [baby-llama2-chinese(0.2B)](https://github.com/DLLXW/baby-llama2-chinese)
-[H] [chatlm-mini-chinese(0.2B)](https://github.com/charent/ChatLM-mini-Chinese)
-
-```text
-[Q]: Do you know about the Yangtze River?
-[A]: The Yangtze River, located in southeastern China, is the longest river in China, stretching over 6,300 kilometers and flowing through 11 provinces. It is China's "mother river." The history of the Yangtze River dates back to the Shang Dynasty around 2000 BC. During the Shang Dynasty, the economic development of the Yangtze River reached its peak, making it one of the most important agricultural and industrial centers in China. The economy of the Yangtze River Basin mainly relies on agriculture, with Yangtze River fisheries occupying a major industry, such as fishing and hunting fish. The water resources of the Yangtze River are also very abundant, including rivers, lakes, and streams. The natural scenery of the Yangtze River is spectacular, with different landscapes in each season: rainy in spring and summer, cold and snowy in winter, and beautiful flowers and fruits in spring and autumn. The Yangtze River is China's economic center, home to many countries and cities, and an important transportation hub and shipping center in the world. At the same time, the Yangtze River is also an important water resource for China, playing a significant role in the country's economic development.
-[B]: Yes, I know about the Yangtze River. The Yangtze River is located in central China and is the third-longest river in China, stretching approximately 6,300 kilometers and flowing through Jiangxi, Anhui, Jiangsu, Anhui, Jiangsu, and Anhui Provinces.
-[C]: Of course, the Yangtze River is the longest river in China, stretching approximately 6,300 kilometers. It is famous for its rich water resources, diverse aquatic flora and fauna, and unique geographical location.
-[D]: Yes, the Yangtze River is the longest river in China, exceeding 3,300 kilometers in length.
-[E]: The Yangtze River is the longest river in China, with a total length of approximately 6,300 kilometers. It originates from Qinghai Lake on the Qinghai-Tibet Plateau and flows through Qinghai, Sichuan, Gansu, Ningxia, Inner Mongolia, Shaanxi, and Shanxi. The Yangtze River Basin is one of the largest agricultural areas in China and also the cultural and economic center of the country.
-[F]: Yes, the Yangtze River is one of the longest rivers in China and the third-largest river in the world. The Yangtze River originates from the Qinghai-Tibet Plateau, flows through 11 provinces, and eventually empties into the East China Sea. The Yangtze River Basin is one of the largest river basins in China, possessing abundant natural resources and cultural heritage, such as the Yangtze River Bridge, the Yangtze Ocean Museum, and the Yangtze Ancient City.
-[G]: It is also known that China is one of the longest cities in the world, with a population exceeding 34 million.
-[H]: Yes, the Yangtze River is the longest river in China and one of the longest rivers in the world. It originates from the Qinghai-Tibet Plateau and flows through nine provinces before finally emptying into the East China Sea. The climate of the Yangtze River Basin is mild with distinct seasons, making it an important agricultural and industrial base in China. The aquatic resources of the Yangtze River Basin are abundant, making it one of the important fishery resources in China.
-
-[Q]: What is the highest mountain in the world?
-[A]: The highest mountain in the world is Mount Everest, located in the Himalayas on the border between Nepal and China. Its elevation is 8,848.86 meters (29,031.7 feet), making it the highest mountain on Earth.
-[B]: Mount Everest is the highest mountain in the world, with an elevation of 8,848 meters (29,029 feet).
-[C]: Mount Everest is the highest mountain in the world, located in the Himalayas with an elevation of 8,848.86 meters (29,031.7 feet). It is located on the border between Nepal and China and is part of the Himalayan range.
-[D]: The highest mountain in the world is Mount Everest, located on the border between Nepal and China, with an elevation of 8,848 meters.
-[E]: The highest mountain in the world is Mount Everest, located in the Himalayas with an elevation of 8,848 meters.
-[F]: Mount Everest is the highest mountain in the world, with an elevation of 8,848 meters.
-[G]: The highest mountain in the world is Mount Everest, located in the Himalayas with an elevation of 8,848.86 meters (29,031.69 feet).
-[H]: The highest mountain in the world is Mount Everest, also known as Sagarmatha, located on the border between Nepal and China in the Himalayas, with an elevation of 8,848.86 meters (29,031.69 feet). It is one of the most famous mountaineering destinations in the world, attracting climbers from all over the globe. The name Mount Everest comes from Tibetan, meaning "the goddess's forehead," and it consists of two different peaks, one located in Nepal and the other in China. Its height is determined by measuring its altitude because it is determined by measuring the average height of the Earth. Although Mount Everest is the highest mountain in the world, it is not reachable from any other place on Earth. Other places on Earth, such as the Andes in South America, are also the highest mountain ranges in the world, but they are not the highest mountains on Earth. In summary, Mount Everest is the highest mountain in the world, but it is not reachable from any other place on Earth. It is the highest mountain range on Earth, but it is not reachable from any other place on Earth.
-
-[Q]: Who proposed the law of universal gravitation?
-[A]: The law of universal gravitation was proposed by Newton in the 17th century. It is one of Newton's three laws. Newton's gravitational law revealed the essence of the law of universal gravitation and explained how celestial bodies such as planets, satellites, and comets interact with each other. This law was proposed by Isaac Newton in 1687. Through analysis and experiments on the law of universal gravitation, he successfully explained the gravitational interactions between celestial bodies, thereby explaining the phenomenon of the Earth orbiting the Sun.
-[B]: Universal gravitation (Human Policy) is one of Newton's three laws of mechanics, used to describe the interaction force between two objects. It consists of the law of universal gravitation and Newton's first law (the law of inertia). Universal gravitation occurs due to the interaction between two objects, and factors such as their mass, distance, speed, and direction determine their interaction. This interaction force is: the product of masses: the product of masses is the result of the law of universal gravitation and Newton's second law. The product of masses: the product of masses is a physical process that is the result of interactions between objects, and for other objects, their mass can be expressed as the sum of the masses of the two interacting objects, i.e., gravitational acceleration a.
-[C]: Universal gravitation is a physics concept proposed by Newton in 1687. It describes the laws of how gravitational forces interact between objects.
-[D]: Universal gravitation was proposed by the British physicist Newton. He believed that all objects generate gravity, which is determined by the object's mass and distance. This theory reveals the relationship between mass and distance and proves the essence of universal gravitation.
-[E]: Universal gravitation was proposed by Newton in 1687. It is a fundamental physical phenomenon in the universe, describing the interactions between objects. This force is called universal gravitation, which exists between all objects, and this force is related to the mass and distance of the objects. Universal gravitation is one of the most basic forces in nature, influencing the motion and changes of objects and having a significant impact on the structure and evolution of the universe.
-[F]: Universal gravitation was proposed by the British physicist Isaac Newton in the 17th century.
-[G]: Gravity is a force in the universe composed of mass or energy. According to Newton's law of universal gravitation, the gravitational force between two objects is directly proportional to their mass and inversely proportional to their mass. Therefore, Newton's law of universal gravitation states that the gravitational force between two objects is directly proportional to their mass and inversely proportional to their mass. This law has wide applications in physics, astronomy, and physics.
-[H]: Universal gravitation was proposed by the British physicist Isaac Newton in 1687. In 1687, he published a famous paper titled "Mathematical Principles of Natural Philosophy," in which he proposed the law of universal gravitation, stating that there is a gravitational force between any two objects, and the magnitude of this force is directly proportional to their masses and inversely proportional to the square of the distance between them. This law is called the law of universal gravitation, describing the relationship between mass and distance. It shows that the gravitational force between two objects is directly proportional to their masses and inversely proportional to the square of the distance between them. This law is crucial for explaining many phenomena and interactions in the universe, such as planetary orbits around the Sun, gravitational lensing effects between stars, etc. It is also widely applied in astronomy, physics, and engineering, providing an important theoretical foundation for understanding the nature and evolution of the universe.
-
-[Q]: Do you know what the main food of giant pandas is?
-[A]: Giant pandas primarily feed on bamboo, but they also eat other plants, insects, small mammals, and fish. Their digestive system is similar to other bear species, capable of digesting bamboo. Pandas have sharp teeth suitable for chewing bamboo, so they are often referred to as "bamboo-eating animals."
-[B]: Giant pandas are mammals that feed on bamboo. They are carnivores. Their main food is bamboo, especially bamboo rich in cellulose and minerals, which does not directly enter the digestive system. The intestines of giant pandas are mainly composed of bamboo, and they need to digest cellulose and other fibers in bamboo to support their normal feeding.
-[C]: Giant pandas primarily feed on bamboo, but they also eat other plants, including leaves, buds, insects, fruits, vegetables, and small mammals. Giant pandas are a species unique to China, belonging to the bear family, mainly distributed in Sichuan, Shaanxi, and Gansu provinces.
-[D]: The main food of giant pandas is bamboo. They usually eat bamboo, consuming about 30 kilograms of bamboo daily.
-[E]: Yes, giant pandas primarily feed on bamboo. They need to consume large amounts of bamboo daily to sustain life, eating between 40 to 80 kilograms of bamboo per day. Additionally, they also eat some other plants, such as grass, fruits, nuts, and small mammals.
-[F]: Giant pandas primarily feed on bamboo but occasionally consume other plants and animals. Additionally, giant pandas eat some wild fruits, small animals, and plants, but these do not constitute their primary food source.
-[G]: The main food of giant pandas is bamboo.
-[H]: Giant pandas are herbivorous animals that primarily feed on bamboo. Their intestinal structure is suitable for digesting bamboo with high fiber content, but they also eat some other plants, such as herbaceous plants and small mammals. The intestinal structure of giant pandas also adapts to high-fiber bamboo, allowing them to digest large amounts of cellulose, helping them survive in the wild. The intestines of giant pandas also contain an enzyme called "cellulase," which helps break down cellulose, aiding in the digestion of large amounts of water and nutrients. Additionally, the intestines of giant pandas contain a protein called "fibroin," which helps maintain energy levels in the body, aiding in their survival in the wild. Overall, the main food of giant pandas is bamboo, but they also eat some other plants, such as herbaceous plants and small mammals.
-
-[Q]: Why is seawater salty?
-[A]: Seawater is salty mainly due to its chemical composition. Seawater contains large amounts of minerals and salts, with the main components being sulfates, carbonates, magnesium ions, and potassium ions. These substances precipitate in seawater at high concentrations, forming seawater. Specifically, the minerals and salts in seawater mainly come from rocks, soil, organisms, and human excrement. Rocks are a component of seawater, mainly composed of minerals such as quartz, feldspar, and mica. Soil contains large amounts of chemicals such as nitrogen, phosphorus, and potassium, which seep into the soil, forming soil. The salt in living organisms mainly comes from pigments like chlorophyll in plants, which absorb energy from sunlight and convert it into usable energy for organisms. The salt in human excrement mainly comes from minerals and carbonates, which have higher concentrations in water, leading to an increase in dissolved mineral concentration, forming salt in water bodies. Therefore, seawater is salty due to its chemical composition and structural characteristics. The minerals and salts in seawater mainly come from rocks, soil, organisms, and human excrement, and the accumulation and excretion of these substances cause seawater to taste salty.
-[B]: Seawater is salty because it contains various substances, including sodium chloride (NaCl), sodium chloride (NaCl), carbonates (such as sulfates), sodium chloride (NaCl), and water. Sources of salty substances: Salt is a common source of saltiness, containing minerals such as potassium, calcium, magnesium, and potassium. These minerals usually come from groundwater, groundwater, or soil, and their content is much lower than pure water. Reasons for saltiness: The saltiness of salt mainly comes from tiny particles dissolved in water. These tiny particles are mainly transmitted through plant roots and microorganisms inside animals, such as insects, small mammals, birds, and certain insects. In summary, seawater is salty because it contains large amounts of salts, including minerals, rocks, salts, and minerals, and the high concentration of these substances in seawater significantly affects the saltiness.
-[C]: Seawater is salty mainly because it contains various salts, with seawater being one of the main components. When seawater contains large amounts of salt, the salt dissolves in seawater, creating a salty taste. The salts in seawater mainly come from dissolved salts, evaporated salts, and frozen ice.
-[D]: Seawater contains many salts, which can lead to an increase in the salinity of seawater. The salts in seawater mainly come from the salts in seawater, which can be obtained from the salts in seawater. Oxides and salts in seawater can form hydrogen bonds with the salts in seawater, increasing the salinity of seawater.
-[E]: Seawater is salty because it contains large amounts of salts, including chloride ions and chloride ions. These ions form large amounts of salts in the ocean, known as salts. Seawater contains many salts dissolved in water, which gradually evaporate as seawater rises, eventually leading to the evaporation of salts in seawater. These salts are completely consumed in seawater and then excluded from seawater. Additionally, seawater contains many other salts, but these salts are different because their floating and sinking processes in water are different. The salts in seawater are extracted from rocks at the bottom of the sea, and these salts can be utilized in seawater. Overall, seawater is salty, which is why seawater contains many salts and salts.
-[F]: Seawater contains many dissolved salts, the most important of which is sodium chloride. When salt is added to water, the molecules of salt react with water molecules to form sodium hydroxide, which is a salt.
-[G]: Because water is essential for the survival of many organisms, it can absorb and release carbon dioxide, release oxygen, maintain water vapor in the atmosphere, and maintain the water cycle. Salts and chemicals in water release oxygen, allowing organisms on Earth to breathe. Additionally, water has many other functions, such as maintaining the water cycle, regulating climate and environment, etc. Water is essential for the survival of many organisms and is also an important basis for many natural phenomena.
-[H]: The main reason seawater is salty is that it contains large amounts of dissolved salts, the most important of which is sodium chloride (NaCl). Sodium chloride is a very common salt found in most parts of the Earth, including oceans, lakes, rivers, and groundwater. When water evaporates from the ocean, it takes away some of the salt, making seawater salty. Additionally, seawater contains other dissolved substances, such as sulfates, magnesium sulfate, calcium sulfate, magnesium sulfate salt, magnesium sulfate potassium, magnesium sulfate sodium, magnesium sulfate calcium, magnesium sulfate magnesium calcium, magnesium sulfate magnesium calcium, magnesium sulfate magnesium calcium, magnesium sulfate magnesium calcium, magnesium sulfate magnesium magnesium calcium, magnesium sulfate magnesium magnesium calcium, magnesium sulfate magnesium magnesium calcium, magnesium sulfate magnesium magnesium calcium, magnesium sulfate magnesium magnesium calcium, magnesium sulfate magnesium magnesium calcium, magnesium sulfate magnesium magnesium calcium, magnesium sulfate magnesium magnesium calcium, magnesium sulfate magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium magnesium magnesium magnesium magnesium calcium, magnesium sulfate magnesium magnesium magnesium magnesium
-
-[Q]: Write a Python quicksort code.
-[A]: Okay, here is the Python quicksort code: def quick_sort(arr): if len(arr) <= 1: return arr else: pivot = arr[len(arr) // 2] left = [x for x in arr[1:]] right = [x for x in arr[2:] if x < pivot] return quick_sort(left) + [pivot] + quick_sort(right) # Test code arr = [3, 6, 8, 10, 1, 2, 1] print(quick_sort(arr)) This code first checks whether the length of the array is less than or equal to 1. If so, it sorts in ascending order. Then, for each element in the array, it recursively sorts until reaching the end position. If the length of the array is odd, it sorts in descending order. Finally, the average time complexity is O(n log n), where n is the length of the array.
-[B]: Here is an implementation of the quicksort algorithm using Python: def sort_array(arr): # Generate a random permutation sorted_array = sorted(arr) # Return the sorted list return sorted(arr) # Example arr = [3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5] # Call the function sorted_array = sort_array(arr) print("Original array is:", sorted_array) This code first generates a random permutation, then sorts based on the length of the list. Finally, it returns the sorted list. Note that this code uses Python's built-in sorted function and the built-in sorted() function, which are faster than directly using sorted() under a time complexity of O(n log n).
-[C]: Here is a simple sorting algorithm suitable for small arrays, including those with lengths less than or equal to 5. def quick_sort(arr): if len(arr) <= 1: return arr else for _ in range(2, 51): if arr[len(arr) - 1:] is not None: arr[len(arr) - 1] = arr[len(arr) - 1] arr[len(arr)] = arr[len(arr)] return quick_sort(arr) # Example arr = [3, 5, 1, 4, 1, 2, 1] sorted_arr = quick_sort(arr) print("Sorted array is:", sorted_arr) ### Code Explanation: 1. **Function Definition**: The quick_sort function accepts an array arr as a parameter. 2. **Loop**: For elements in arr, recursion is used, traversing the array from right to left and swapping the current element with the current element in the array until the array is empty or the array length is less than or equal to arr. 3. **Recursive Call**: Traverse the array, swap the current element with the current element in the array until the array is empty or the array length is less than or equal to arr. 4. **Check Array Length**: Use if __name__ == "__main__" to check if the array length is less than arr. If so, move the array from the larger array to the end of the array. 5. **Output Sorted Results**
-[D]: As an AI language model, I cannot write Python programs. However, Python is a very popular programming language with many excellent libraries and frameworks, including NumPy, Pandas, Matplotlib, Pandas, Bandke, Norm, and NKlearn. Python has many excellent natural language processing libraries and frameworks that can handle and analyze various types of data, such as text, images, audio, and video. Python can also be used to create various applications, such as web search, game development, scientific computing, etc. Python is very flexible and supports multiple programming styles, making it easy to find solutions that suit your needs.
-[E]: As an AI language model, I can answer your questions. But I cannot perform programming operations because I cannot determine which programming language you are asking about. Please provide more specific questions so I can better answer your queries.
-[F]: Quick Sort is a simple sorting algorithm that divides data into groups, sorts each group according to certain rules, and ultimately obtains a sorted array.
-[G]: Okay, here is a Python quicksort code ```python sorted_print(sorted(sorted(sorted(sorted_print))``` Output: ```[1, 2, 3, 4, 5, 6, 7, 8, 9]```
-[H]: Quicksort is an efficient sorting algorithm based on the divide-and-conquer approach, with a time complexity of O(n log n). Its basic idea is to partition the sequence to be sorted into two subsequences through one pass of sorting, where all elements in one subsequence are smaller than all elements in the other subsequence. Then, the two subsequences are recursively sorted. Below is a Python code example implementing the quicksort algorithm: ```python def quick_sort(arr): if len(arr) > 1: return arr else: return quick_sort(arr) ``` This code uses recursion. First, it checks whether the length of the array is less than or equal to 1. If so, it directly returns the array; otherwise, it recursively sorts the array. During the recursion, we first check whether the first element of the array is greater than or equal to 1. If so, compare the elements in the array with the first element. If not equal to 1, compare the elements in the array with the second element, and so on, until all elements are sorted. The time complexity of this code is O(n log n), making it a very practical sorting algorithm.
-
-[Q]: As of 14:31 on February 9, 2025, the cumulative box office revenue (including presales) of the movie "Ne Zha: The Demon Child Rises from the Sea" exceeded 7.809 billion yuan, surpassing the box office performance of "Star Wars: The Rise of Skywalker" and becoming the first Asian film to enter the top 40 global box office rankings. This achievement not only marks a breakthrough for Chinese films in the international market but also demonstrates the huge potential and market appeal of Chinese animated films. Summarize this passage in 20 words for me.
-[A]: The movie "Ne Zha: The Demon Child Rises from the Sea" surpassed 7.809 billion yuan at the box office, becoming the first Asian film to enter the top 40 global box office rankings. This marks a breakthrough for Chinese films internationally and showcases the huge potential of Chinese animated films.
-[B]: On February 9, 2025, Chinese films broke through 7.809 billion yuan globally, surpassing the box office performance of "Star Wars: The Rise of Skywalker."
-[C]: The movie "Ne Zha: The Demon Child Rises from the Sea" surpassed 7.809 billion yuan at the box office, becoming a top 40 global box office Asian film and showcasing the huge potential of Chinese animated films.
-[D]: "Ne Zha: The Demon Child Rises from the Sea" surpassed 7.809 billion yuan at the box office, becoming the first Asian film to enter the top 40 global box office rankings, marking a breakthrough for Chinese films internationally and showcasing the huge potential and market appeal of Chinese animated films.
-[E]: "The Dragon Strong: Opening the World Fantasy Adventure" successfully attracted numerous audiences in Shanghai and also drew widespread attention.
-[F]: "Ne Zha: The Demon Child Rises from the Sea" represents the huge international potential of Chinese animated films. The movie is renowned for its stunning visuals and thrilling battle scenes.
-[G]: "Which Is Worth It" was released on January 9, 2021, at 14:31. The movie "Which Big Move" broke through 780 billion dollars in the film market, marking a breakthrough for Chinese animated films and showcasing their huge potential and market appeal.
-[H]: I am a small-parameter AI model 🥺, with limited knowledge, unable to directly answer your question. Try another question 👋
-```
-
----
-
-🙋Directly give all the questions and the model's answers above to DeepSeek-R1, let it help comment and rank with
-scores:
-
-
-
-Specific comments
-
-### Scoring Criteria:
-
-- **Accuracy**: Whether the answer is correct and free of obvious errors.
-- **Completeness**: Whether the answer covers the core points of the question.
-- **Logic**: Whether the answer is structured and logical.
-- **Code Quality**: Whether the code runs correctly and the logic is clear.
-
-### Review:
-
-1. **Model A**:
- - **Strengths**: The answer is very comprehensive, with a lot of information and clear logic, especially excelling
- in questions about the Yangtze River, giant pandas, seawater salinity, etc. The code has minor flaws, but the
- overall approach is correct.
- - **Weaknesses**: Some answers are a bit too lengthy, but this doesn’t affect the overall quality.
- - **Overall**: Best overall performance, scored the highest.
-
-2. **Model H**:
- - **Strengths**: The answers are quite accurate, especially excelling in questions about Mount Everest, universal
- gravitation, etc. Although the code is not fully presented, the explanation is detailed.
- - **Weaknesses**: Some answers are a bit verbose, but the logic is strong.
- - **Overall**: Second to Model A, stable performance.
-
-3. **Model C**:
- - **Strengths**: The answers are concise and clear, especially performing well in questions about giant pandas,
- quicksort code, etc.
- - **Weaknesses**: Some answers are a bit brief and lack in-depth explanations.
- - **Overall**: Good overall performance, but lacks the detail of Models A and H.
-
-4. **Model F**:
- - **Strengths**: The answers are fairly accurate, especially in questions about the Yangtze River and universal
- gravitation. The code is logically sound.
- - **Weaknesses**: Some answers lack depth, and the code has a few small issues.
- - **Overall**: Average performance, with room for improvement.
-
-5. **Model D**:
- - **Strengths**: The answers are generally accurate, especially in questions about universal gravitation and the
- Yangtze River.
- - **Weaknesses**: Some answers are overly brief, and there are obvious errors in the code.
- - **Overall**: Average performance, needs improvement in code.
-
-6. **Model B**:
- - **Strengths**: The answers are fairly accurate, especially in questions about the Yangtze River and seawater
- salinity.
- - **Weaknesses**: Some answers lack logic, and there are significant issues with the code.
- - **Overall**: Mediocre performance, needs further optimization.
-
-7. **Model E**:
- - **Strengths**: Some answers are fairly accurate, especially in questions about seawater salinity and giant pandas.
- - **Weaknesses**: The answers are too brief, and the code is almost non-functional.
- - **Overall**: Poor performance, needs significant improvement.
-
-8. **Model G**:
- - **Strengths**: Nearly no apparent strengths.
- - **Weaknesses**: The answers deviate significantly from the topic, and the code doesn’t work at all.
- - **Overall**: Worst performance, needs major improvements.
-
----
-
-### Summary:
-
-- **Model A** performs the best overall, especially excelling in complex questions with high accuracy and logic.
-- **Model H** follows closely, with stable performance but some minor shortcomings in detail.
-- **Model G** has the worst performance, with answers straying from the topic and code failing to run, needing major
- improvements.
-
-
-
-### Scoring Rank
-
-| Rank | Model | Accuracy (30 points) | Completeness (30 points) | Logic (20 points) | Code Quality (20 points) | Total (100 points) |
-|------|-------|----------------------|--------------------------|-------------------|--------------------------|--------------------|
-| 1 | A | 28 | 29 | 19 | 20 | 96 |
-| 2 | H | 27 | 28 | 18 | 20 | 93 |
-| 3 | C | 26 | 27 | 18 | 18 | 89 |
-| 4 | F | 25 | 26 | 17 | 18 | 86 |
-| 5 | D | 24 | 25 | 17 | 16 | 82 |
-| 6 | B | 23 | 24 | 16 | 15 | 78 |
-| 7 | E | 22 | 23 | 15 | 14 | 74 |
-| 8 | G | 10 | 12 | 10 | 10 | 42 |
-
-
-### 👉 Subjective Effect Summary
-
-My personal evaluation aligns with DeepSeek-R1's results，and：
-
-* The ranking of the MiniMind series is very intuitive. The larger the parameters and the more training data, the higher
- the score, and hallucinations and errors are less noticeable than with smaller models.
-* Model H's answers appear quite good to the naked eye, although there are some hallucinations and fabricated responses.
-* Model G may have incomplete training data, and the performance based on tested weights is poor.
-* Repeating the timeless Scaling Law: The larger the parameters and the more training data, the stronger the model's
- performance.
-
----
-
-## Ⅲ Objective Benchmark
-
-Now, onto the much-anticipated benchmark testing phase. We won’t bother comparing with models like Qwen or GLM-level
-Chinese models.
-Instead, we'll focus on a selection of <1B micro-models for a comparative analysis.
-The test sets chosen include C-Eval, CMMLU, A-CLUE, and TMMLU+, which are pure Chinese language leaderboards.
-
-
-Evaluation Framework
-
-The evaluation framework chosen is [lm-evaluation](https://github.com/EleutherAI/lm-evaluation-harness),
-which is very easy to set up and run after installation:
-
-```bash
-lm_eval --model hf --model_args pretrained=,device=cuda,dtype=auto --tasks ceval* --batch_size 8 --trust_remote_code
-```
-
-
-
-PS: In these multiple-choice-based evaluations, to avoid issues with inconsistent response formats,
-the common approach is to extract the prediction probabilities for the four options ('A', 'B', 'C', 'D'),
-and calculate the accuracy by comparing the letter with the highest probability to the standard answer.
-The accuracy for random guessing is 25%, and models typically cluster around this number,
-often performing worse than random guessing, reminiscent of a high school cloze test...
-The MiniMind model, with its modest pretraining dataset and lack of fine-tuning on the test set,
-is mainly for fun, so take the results lightly:
-
-| models | from | params↓ | ceval↑ | cmmlu↑ | aclue↑ | tmmlu+↑ |
-|-------------------------------------------------------------------------------|---------------|---------|--------|--------|--------|---------|
-| MiniMind2 | JingyaoGong | 104M | 26.52 | 24.42 | 24.97 | 25.27 |
-| MiniMind2-Small | JingyaoGong | 26M | 26.37 | 24.97 | 25.39 | 24.63 |
-| MiniMind2-MoE | JingyaoGong | 145M | 26.6 | 25.01 | 24.83 | 25.01 |
-| [Steel-LLM](https://github.com/zhanshijinwat/Steel-LLM) | ZhanShiJin | 1121M | 24.81 | 25.32 | 26 | 24.39 |
-| [GPT2-medium](https://huggingface.co/openai-community/gpt2-medium) | OpenAI | 360M | 23.18 | 25 | 18.6 | 25.19 |
-| [TinyLlama-1.1B-Chat-V1.0](https://github.com/jzhang38/TinyLlama) | TinyLlama | 1100M | 25.48 | 25 | 25.4 | 25.13 |
-| [SmolLM2](https://github.com/huggingface/smollm) | HuggingFaceTB | 135M | 24.37 | 25.02 | 25.37 | 25.06 |
-| [Aquila-Instruct](https://www.modelscope.cn/models/BAAI/Aquila-135M-Instruct) | BAAI | 135M | 25.11 | 25.1 | 24.43 | 25.05 |
-
-
-
-# 📌 Others
-
-### Inference and Export
-
-* [./scripts/convert_model.py](./scripts/convert_model.py) can convert models between torch/transformers.
-
-* MiniMind's HuggingFace collection link:
- [MiniMind](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5)
-
----
-
-### Based on MiniMind-API Service Interface
-
-* [./scripts/serve_openai_api.py](./scripts/serve_openai_api.py) provides the simplest chat interface compatible with
- the OpenAI API,
- making it easy to integrate your model into third-party UIs such as FastGPT, OpenWebUI, Dify, etc.
-
-* Download the model weights
- from [Huggingface](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5). The file
- structure is:
- ```
- (root dir)
- ├─
- | ├── config.json
- | ├── generation_config.json
- | ├── LMConfig.py
- | ├── model.py
- | ├── pytorch_model.bin
- | ├── special_tokens_map.json
- | ├── tokenizer_config.json
- | ├── tokenizer.json
- ```
-
-* Start the chat server:
- ```bash
- python serve_openai_api.py
- ```
-* Test the service interface:
- ```bash
- python chat_openai_api.py
- ```
-* API example, compatible with OpenAI API format:
- ```bash
- curl http://ip:port/v1/chat/completions \
- -H "Content-Type: application/json" \
- -d '{
- "model": "model-identifier",
- "messages": [
- { "role": "user", "content": "What is the highest mountain in the world?" }
- ],
- "temperature": 0.7,
- "max_tokens": 512,
- "stream": true
- }'
- ```
-
-# 📌 Acknowledge
-
-> [!NOTE]
-> If you find the `MiniMind series` helpful, feel free to give it a ⭐ on GitHub.
-> Due to the length of the content, mistakes are inevitable; please feel free to report issues or submit a PR to improve
-> the project.
-> Your small support is the driving force for continuous improvement of this project!
-
-## 🤝[Contributors](https://github.com/jingyaogong/minimind/graphs/contributors)
-
-
-
-
-
-
-
-
-
-
-
-
-## 😊Acknowledgments
-
-@ipfgao:
-🔗Training steps record
-
-@chuanzhubin:
-🔗Line-by-line code comments
-
-@WangRongsheng:
-🔗Large dataset preprocessing
-
-@pengqianhan:
-🔗A brief tutorial
-
-@RyanSunn:
-🔗Inference process learning record
-
-@Nijikadesu:
-🔗Decompose project code in an interactive notebook format
-
-
-Reference Links & Thanks to the following excellent papers or projects
-
-- No specific order of ranking
-- [https://github.com/meta-llama/llama3](https://github.com/meta-llama/llama3)
-- [https://github.com/karpathy/llama2.c](https://github.com/karpathy/llama2.c)
-- [https://github.com/DLLXW/baby-llama2-chinese](https://github.com/DLLXW/baby-llama2-chinese)
-- [(DeepSeek-V2)https://arxiv.org/abs/2405.04434](https://arxiv.org/abs/2405.04434)
-- [https://github.com/charent/ChatLM-mini-Chinese](https://github.com/charent/ChatLM-mini-Chinese)
-- [https://github.com/wdndev/tiny-llm-zh](https://github.com/wdndev/tiny-llm-zh)
-- [(Mistral-MoE)https://arxiv.org/pdf/2401.04088](https://arxiv.org/pdf/2401.04088)
-- [https://github.com/Tongjilibo/build_MiniLLM_from_scratch](https://github.com/Tongjilibo/build_MiniLLM_from_scratch)
-- [https://github.com/jzhang38/TinyLlama](https://github.com/jzhang38/TinyLlama)
-- [https://github.com/AI-Study-Han/Zero-Chatgpt](https://github.com/AI-Study-Han/Zero-Chatgpt)
-- [https://github.com/xusenlinzy/api-for-open-llm](https://github.com/xusenlinzy/api-for-open-llm)
-- [https://github.com/HqWu-HITCS/Awesome-Chinese-LLM](https://github.com/HqWu-HITCS/Awesome-Chinese-LLM)
-
-
-
-## 🫶Supporters
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-# License
-
-This repository is licensed under the [Apache-2.0 License](LICENSE).
-
diff --git a/images/1-wiki.png b/images/1-wiki.png
deleted file mode 100644
index 5eba0fc195eaff7477f4fb21edafeed758755d4c..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001
literal 139475
zcmdq}hd^slbyZy&WU86_RPpAPGp7C-ru|H
z_5S?+h+ntcb-Q)BxSr?p@wkuUejLYLpn{w@E;cDP1_lPMq{Qnt7#P>1;UA8hSK%ws
ziu7MFFy3QGz7|$?)mfhj&>~d(g|)TGszG2_RQ_1iYJv>QBM{3M+d`Quh+I^3sEw3B
z>-AF$t_vh_79dOQzkjr`|Djy`j46~
z9}a&8YcCd>|Mypn6k+>o|NEi<74QGP)+qA-2Ve3~OpM$<{?j^tav*JB*i%v26C5lZ
z6{REOuI#uV_9vFnkC=rpSs>nLN#;CX$*rZOJ^+9D(X)t37u)_c9~0ZT)M_WjGhUD7
z{^al9lbg@Gaj`Q#A`Cmxa>-xa-IZG=D>3!;{o+}O`~yY^?$0%y9k=+752QA_?G;cx
zN#|nT*dWEhIpR^t;d*hlH-#7n?SzIzH(WMwKPBLH5
zmlfK*>VQwN#QF%sg8uRb-^;%{_&398+W(N>(QLUtg^btcHvGADtKRFx`PHkdtlW`F
zNlD4cr|89=0DOwsI=hA8Y-#vL0k31ncki}Zd~d+dpDd^O*!-#qO?966=5&63-qh5z
zvpkT-ZTWLFU+Km3=N)0>0>fQ>eWB;^-h2*Ay>2^8%sL3yBlO~KL|+o`NObZuyLos*
zWMt$(nvlG#EHeXxI~pZvW=6-HJ^!aQpw?=#Lci7jh5IfEug$|^19)a&V4$a#wZ^|D
zuk!x>9KPI|bE?~ipX0b68)*jbi!NBl*h{+=))Ldizi?p{;xA}Yve{bVgtx~QW
z9v+^q&lz{=&xF}Vk5cUho~KWJo9&a(OGi7dYHA}^N^3rFtc=fmQh9G+jI#W9k}82A
zArHCC?%?xR6Z#DhF{)E$W&sd*aa9!OTidPe`v
zx%l^-&raB*zFW6@)$2J*#PLT=4Tb7&;BARrd9QgLiThnAD?59p-T@B%{MOBz6Yzf9
z3!SC9&5y{)q_wMV;6M8G>C@n*TB**)!UCsZ$8CK4@W`j)Z)}~uzrWJxaqtgb8gB4c
zwdHt$O5qY*fX%eJ;D?vmFOFp^P5LxztR`!$wPc5>k23#SoAdrC8-Oa5p^@Yb!hGCxO8D;GySr*!)>B+l%_;m&7E@L9XVjiMeZ1uSj$c#c
zJdWA|3F_$8n8b4dzk$LbrTxDBYP4_sm7)<^4V0
zOV)1S7%xhH8_(}xQjtqb{q6Al%uv9^M4o7E{h^TF%XybMYWZY|*p*;n(<05-YPX$V
zyVX8Ev)c=p>FJj#J@akvczc1jR3|nkD`s(VTg5vgqfl)mo|)528!f-R@#|C{LqQ7y
za@2lw{NUgqx2?Uw<+6?2UXj&K
zh35W|K2fwXbO?JtpihLg6DrjIXLM?JFy)s;Y=tb*3Tsj`!BW
zDTL%CB+6ManBda-`}0gjv3V*M
zqL;9$q=-u8v7SP>?U=RlSWT39o$Oz`ftUQ;4GqP^WplQ8{tW$hDAW2^jnm)356yDE
z`U@SQ2u&YIEp}YCd0PF!ubR=5Mz%UlG7q`CclU`+9AXPRT
zy|Vi_vr$B!TXx;}vXS?Iv!r|anGFflShEiU>61lUeh1*Voct^GW`c_cTI
zBbzIi;(oZLPu4i{U3GBo_6E)B{2<=ONN#v^v{E#UY~C4^@#zxn2AxKCMe(5X=YR42
zh2G5N`H0O_KN0cY5-OVdJnJqgDT%n{7_#3FS$5P@hQB-1nIsY)_UHwiuS3(G7y+%^
zG~{XKlmI@5m*bbzH2hn){y4d#kB6R){LMIRbJ|RGLEZFu$^0fREse{$Xay%XEowqt
z2N$DeKsB1ouL{bO+t%-v(N>erFPi?8)0L(Jhil^$*u7AEGDp7|v|~#lVr)}VQpmV1
z+H?n&EhfsIv@@*A@$vIRNvjbFz+0DNI5zw}CzK=}zQ~yO=InSczm}YsxKp?0oJphN
zlWimU*7kOJ;OYMASfS0#x9#Fu+TA}2Z{PO*5|#g1q|wSdrT5Ox?$+(wz5V@5W7Uf8
z?))dS0fOF{85yoAf-yO{No2K8Cr2uKPt7|tGZA&qCRs&-4zj
zt*x!M6|pRbD_usRwwB6j3e`$kbelZeKHs&N&dZed_V)g^(MU~0<9)JG>lrLGTA;#+
zT>@#mkRclUvqZax6{T4l_Hi!c0q6A04E-&ajj8G&GCe`Ev+64I5&u+ANSjQtQ1sT;
zSME2vn)UY7bEFHNsu1)k0xnAf1{qKX_=Py%t^F*Pnxn@h<=WcaeQuAsc@jp$$9J^9
z@$2IJWb^#&1g^Xeayo(2^s=h>;HMKE7k&|N`3GmSNdDUAzX*M1XRue=LoVcvIOT9!
z91yxdC0VlVZqCi&vHIu{2M>EFq%HJi^(UK?YW}6|xzB>GBE+`MiI?rDven(WqKNU*
z){4H~da^=PFk&L9$?L?#*mzs4-2cm$FVO3i-Niq>_`0?8(uW`wnnl^LE!W?8!eP8e
z&q*FU&}i3Yp-@p)j){+thc+J-6%`S&1s^RhFAox9rokorDpKX?aVPxETsl?I>lo_Q
z#DqF_bbfw5)Qa7SjKmpsU3GQWrQQU94-g}0V*2#AEfo3qgom=MuN>aJd**%mjI~;0
z_l6js!}feTfCatMnDnkrilS>!SDjWz^^>GdBxorRq>S|RE90eU(}awwMZZOVs(Nmw
znww;%okr8TnOm#fBPJdxA-4=je&OLtjxq7_|u9)a(
zPM3dgn>6-wg?+JNC|%do*op{BdQ0523_29_V(9L^{Ae6x5;o9&$#>EFPbrJLk;O{_Ets$&Z$(3)SuLt0ziZ`NOL*gf?D#&+K9R*j(LI1lsf@`dNt$U6uGpZ(<`cPl$Eo3
z#XCbD@$dzim_8mJd*xV*PEe&j4+M&ah>29Sl0ojp+C7kdQ;i
zsv8u~U0l8q`xU7!S2Q9`r|0e89J)Dw4>Sl2T>&3dE*z#+US<3{qu(Z7aG_|NOJjY+o2il2%@
z87=J~zjlckN!wal(zCKY5o<{PDAQ-k&N?|pQwr?RFa(KD&iO2hV(9farcCE`j+E((
ztgs!YMn{hT3Ou}KaaZcDN{G+ld^^rL2Dh<^$s36!vh98-Y*EqCM}}4!uC9lN+Y3i)
zTqKEH=EHz1k9af1!Pw&UY$E&&joEaO7ji1Vfr+4?+Z~Ye5(Z_8+*sT2gN*GTKE2;_;
z2W28T+9HgN!6ea)WCj#FY`2lfYujYliKnAYrik-}jA5|7-A}F1d^Atd6DP!dZ`BDJ
zu8y|Wbc4&l%uGrnN-UJL+>WM*EIK1Ib20SEWzdQ4xBU8*#O+s#*Yw`>U_mixfymkS
zN76K5&}60Q4u7BDr%zjb9Q@`!nQxU>yJH#cE9>g(x3Mj|@NViMH}+Zs5M3;>ZJpu*
z{7tqAam+oAd~|deo{!FGc=GSC
z+9y9h6|XgIX#7WpTvg5+nr?>4DJgRZx7Ofw5lP8g4N6szx%19Sh0p9!pAj#p9zT9h
zJV}6$kN@*Aq)6p){*aU{o6T@mpf?Ln6~7w~j>Utf3DT&}@OVS5<0xvxkFf0iRAyee
zCL0?h*89W}b9)?o{P?i+3r@?QOgg4F=nzmn18e=ZNkw%E!YI5CKa~E`n=q4CT5Rj+
zfaV*_`{C0txlf2HN(-_2v-sq2+fYYezM`T6B~5^F>e45$O-oA}(9ZP(_VnizYXaB{wH}B37$RffaI1nW=N$AhpEx-L80{llu{cGHNpL{R03EQBsqM=aQ}4%g7}?
z)C*{Cv9GLqklVerWS#W6D_iMcH%yd^%;#sN^;ro?^WhW=z7J`dv&
zVl?b*|Cg=(g8CW%@Q+~PMWd+ac9f-Mwu{u%<>UfE`+*c}@6|Bg_YE7`-G`&uu@VQt9U&`%J$c`EC~&>X^<=
zVAR0awM;Z+z+1nV#^27ku5&z`ZBG#U>tNFR{nYDlWHB^VwD1oIXSIp6-Cf5K{c6+R
zFD+|KePd%PI+@jSEGuon#H^^37e+AFFO4yH9B!o}uEfl{`ZWUNE}!O@ot;Iz@&Zs1
zQTt)Ala#eW4aNtfP={acN^%L_iV}dozpt%pDJn)j=l#C*r`64`hC{XZUIJAIGXHeX
zQb+$Fs6;nDT8q8^$+gk@-=uA7ZDykIzWT%I`Z&_C2z_(0EbjTD_}^ib1DA)>u9
z9uCuKw>71i&y#5Y^ozJhmsF)%=8KG`)*<2FU
zWS=4NRf*2B!6jjPStTEUM}BBvjePnpOD3h8_!>Q9u=$K;)ti!~Ip(YHr5xj@QH&`~
zr|2FSM7FU#h)78pG{uIXtmzmjq91Ai$d;9onkU*1;CPw6lNEm1EeeWso4wIk3cX82
z&P>mqsoNxPZN1Y;=NNfnT_pH1!~1QMX99PBGM|HMq-q?Cb~jIsWHim<;v%B|U*8vY
z5iUh2zefhv)b$O)GEv(*1D@<8)Ms0*C#stDQLI=Pzp_eE&S%F?D;PmTje-iQs!P9r
zKd!|`s#vGquVMY>vf0nLJ_fK36{l!+O6@2oCkLZ%+tkpw(O<7^fDl3Z8d)2I78)8F
z24g|C&1Z4MljXz1!~VEr-5VpNxmjR=J6U1jCNqoY6?i&+GpZc(wT+W3KK
z^#0~@a0{Kop2d)q@>x?Z8Y
zgip`T-2BMNxXdD^pJ6TVNitUwBpD5~w)(0~yl|tawipXyRV5(X8PS^)Sx~ir!QcL*
z%4#x7&1@0MJ~}P}dH|a8kZsP<*Af$nXL8cA{F
z#?j95@DPEEipq~&lVj<4mTgMu&tYLdO+ox?KQgNb_zXDf_xO3OTee6lAn
zcblm$EiFY@_}2K4@#qhvzOX-0x8pwnPO3-i6$b~$INd);^B<{P`w?|M}U0ZhauvmscVrFWjpTs}T4**bSW~ib^MO
zGYrpNw_eTy(j?ZoB4%}by-8h!dVAacF@0NYkO|?;H)>u&Og^0}7(5cwclwwTbxX{Fv!ms-;h~ZL+J^ou^wGfO1l+ENT3Xuo
zmM6IlT-q{GLF9OF<@Kev4bHMQbk
zzYRIYQegZ`SC16Ol<_gbBMaBbF7I!KHdh`5T<=!9-fjh=_>qjLJ_-iN`Av
zD;<{Q>eI1taNPW`Ctl)do~um60q56^8=IWm%UDI}W1cr`*2vu%r3Kgky2M`}ttvCS
z#cm><;;T~_LsD7ZXUE3I#s~?5Qi>N**Cbba6A)KJII*!
z&v)EelC4jFqom~KN8MF85q{F#cL>}m%3o^yUCl$cM)x{RG+<;}`PAONzD1m(%Z6hw
zvfaL_Co&HF%Qm)j&%l5@&?VB+FgX6{=?xAF;x~qiYl}Dk(b_6vlLsWFPCC(1Z~;tD
zJMG8cqqKyDKX?vsw*Qu&I!SABluPVXB{e?pD^#<8EV+%ICn9B#m83sBn2S1eZF+fw
z17)%IDnQ
zr`Na{BBi9X_%iCb`8`M$4R7zL1_5gkm==sRwQDrJR;1&-@7U$a{JAU?yHV36Z|mVT
z0Qb0Ik~F&SN!ieG4`2O745Q-`kzQNVC-43B2c-(3o?h`p0oV#gMn<2v4}cnJC)Sb#
zOkqapYnO3XCmoN1f}(sF(hS|sP}YtDrkG|{nTiUy_-sr|vrU@7{gF8x!}`uS=v${Y;&+eZW9!-HeD&0!x;D;R3GB+vL$Glo74Mc0I?aIbmOmUzSS&y
zp`0nw{kcVmPni+&o*NN^?DMS$f~5fksbyts_37tF%M?24iMCBgVnu2_d^xg;K8nf4
znvL$&Q!|~bznZ)n%qY=gYP#FlPL~9TK{l{;3&AQ1?I#}vJBLR`Rx+75Iioj7*Jx_B
zYHb2l-+c1-M^hGesf_{g5W`x-Wry9rOZDr+wHvP^44*%L9uI;cgwXQDQxCt38Q)UeZmh^cB6Dn>ICy|kYDh~gPh!qrB;hH?CmR6MXtOzhF1lF
zLhB(uz@QG|LYV}H07@x6y~j?7scOqytH9|8_mH9P&*wh?8R_8U6dDwCuUV0dJXTcX
zF19@)qXT3S+R{r+3h_g#;BUHS`y~MLpQ^Q`fS}FH%v9eb+}qUo8q)3YdGj;D&Cy~<
z7nkMWKBdy&{4W6fPrMA-Mhn%#)$)8vg^L_fU6G#U*afb-3MNW|eKH~+ouaiQBtDx_
zi@tlu#yTuD*W}f1V#fAFmIx{P&{$9_+Xw6Z_TQnWH6?I3Ta?9AVI)tkO8yv3_bjK;
z5%MBkTR-Kw8~Q*gr?Mr1Fkk45injK8pNA^OSFA>=S^#Z<3s;=#4-IhWSy|=fvx3k!>V9^3PnrFrZ>
z7hm|CKQAqdVsOt>wF5b>K7Ewr*Y}6V4Mb~uTU)<_nPqp;%n}=CU0C1qbejcB6>em6
zm%ov+``@en<*CQS#Dt3mr1hTI+<5S-umLBcvGK@2x^f8Bfn`JqG8{sJ5s1}hd@5h8
z?e@g3$p*DajG6v-Df|vgH;{c&cZo5Wq*;;He+Sd&U4L(G4iIaFkzC)*B;~RCyShqB
zLX!8Ejh>&MvOYT}r#q>*rLLFi$rItC$mbvr$3)~L`(t{4#O=)ZA=jS2x_eyAh#1Al|
zzC<-{k*YvkG4qHtZsPheL}?Kf)D6gz0m39G&Kk5DrBE`*A=c!Lh#@A0Mb^8X70~WZCqpT}@Kyj9qJ?#Dll1ld=D@Atl?INvOmW``Ty|>-%
z_vUvmCtLqx*;ZSf=CggRavfY2B!SQAZj_RRj9ZU#A{SHpZZ}I`RMPXHuUNSvj7*8s
zFew0q^m~5Z;9FOPH>fc#p}RtwOf~fKA4%hjmu9ZZcr@B_=vHNJDB(!rOm$^DN+5X7aKcB17CFv4STfh1v
znW?C#7#Oj>k_;wSDW+l!*Z?|&VC4)iFjcZ16HN06()V`;u2Qbs+fFpaOfCy;~
zq-#!nSQrk-cU)(UnB}+32{q*m=zCsx9(vhV{$_!>;G0wMux3@}!o_e{u(Y`*X(0nE
zy<`IW({IDl)S&|r|JIV&-sySHHSLOAe7#^sIOd=BCE+tLGT5J=Dj|XCh^3G@`V0S)|MR90<8sp{Xhg>g7Dn}vnN@y=o}Gc61tT!go9
za2ANTmsVEjOJ9uztK}C8dt9^zuk>E_mloH_Up^OSo$(3!{3PEnK|YbTp<|XOl}QuS
zE^>ib@hs1td~451jv-R1aRr6S87@nkugt^l&b~rVgJkCAJpf9!D$y|=e@B8uf
z^6skZAUMaH7=4~2o@+R0;q*KtSX)oNIG7{cc7iQL$c&YfAsZTSXel^8f>sUJ^JN3vZqH{r~PNKmYbgiHa2#N
zZnF)GrHY-6uyClu-@zw)$1E~0U%sDKAY_7XxX<#I?q}hmmyHTSxIo?)bxBwc31at|
z%sV?c5HPBk*98wq*nmiD`LjrW!J0v{)<)Ln8^{^+Ub!uJQKf8ZGAaDA#O9)JZ^>if
zJ?tXZ`T^=fyC+F(e_OFklyHL{kfhylA&*{Y_B&D51Nad?{~_1L<7?M-XV3qtNx-9a
z5VKt2%nmj@(j*^}AI-}v0?Z`8^EG6XHI)shdfuf@>wW?6xB0#5<*!aF4*zK%Ng6ab
ztMu0o3875;UpPv*=hSuw2k(TD8>X{^X!rLgGvAVzzq0M-!9ZXk^%V;oZK_(pWEuy0
z3MnYny>^B~eO=E)9c3j4kjo?0%Iy}!08p;h!5@WXL%7-4ns;N!(5qxl?{9ZpIa6*N
zQ}`-l8S7tgxW)4Xw`8-T9!0aXb%#sZ2JKhGpv_F*i80*lO1)|grI1tJMTKVOSz`C=ct
znHv-o)TW%bi*%cB3zQP#WBw?p97}{8C7G_{Z~J`Oji1e`qli0i!n6Gx=th$g4W&Km
z;7!(8tC??GGke0u_4R><9dzP
zBvn@K3}uQR&398#Q*SfpvByoHpP$$Q{RF3Q@j+$;d`6mE1?8tu%+$8AsIE6Em>Jx2F$@~+r8FjNRqQshg(5S$}
zl5V|IgO7Nbo8!43LqcwsSATTh5mRP19cST-j`1AMW?5Wpclp=FTiMtrG4_b&6I#bJ
zP0-di!ZF`yD3pwSmac#-Vt1Glk~Ro!blLqkl0BH1THZ*o9hQq7&5Rzf?yCv-eAAo|k8`sjr_
zC=r{+}qZpxN%kKDvoDBml9e$?#+oPnXqW0!i2W`IlUfB|2Z{`E{G%oRP-
za4Ogg1o5AST2~}d*{d3rCZT53=y6@2ZR|0iRCS7K!L#wd;vK!LstRBsfH+^|k%{wX
z8l}0!CRTtZ{LbrtfB$9!f7)WPKIy$)YTigj0PyiqQIxC|k4NrG9q(`S1Xul_F)8Ab
z)fg0yCXZu#gTC6P&otYivbWtDsFH6xHF
zYokQsSS3Qj0tZPDiuE=vc2gJh3uB!3LM2@$xmmV;IFj+^LMS#!O4&c+?BB=_ljk%D71
z^$GphUx#r`#_m3R$Yd)CdP`|p-YQis_JZv7n8tRe1N+u6avXRK`vy*07v$K%hp*pD9E?2g#NrLOeI
z^T#kbaE|v}JSU~%m8#6}x~`Lj=Qf=PMzNiC1~xC*8deBMX;|L(bE9I>@eyqhe=Zyj{tHQ%~
zS9R~*y9b!6(sAWYxsFu*|1V0SlAhr*dhBR;Rm)d<9N3hLSr1EOZ{oC@Qq?j%G7w(p
z$MLnZzp9fJ3zjXd*{Cgxx9(~LV=^{DAU^Kn-fi?^(Lv-*|CFZapPC*ijNtVJyIqW$
znO|t48ULuDIMQNS&P^h!?$HbTtJivPuw9Lbx_d^>PGj`M?uM~??)d*mx~Ybi2RxHs
zY0{Vz)|b1F+ubwwS~T0$4XKsNW+cHaZiGBOb8+L0nx1jHWa_l3ctNW?a@{I(cOlbK
zaEc;ZWHmOOfMV^vbmw7m`NY%c`lhvula*#>-FL$Dn>VB){^u=mJSs=2s8Y>{uExPKk863o5
z_@&;9^!|M@3OlR?=FC;EuyQ*M+zx_u7)`-v6;<2jXhqdi;B6l*UwvzLKYgGB23E4(
z5LDKq8tN%ecE$V{^AVIxlvd+{o(5<)iQE*oPyV1{$}Lp51J3AVrIo~Z_2Z?w&4>tZ
z5_E>-Q~luMH#tR1I}Pl`#u6n9r9JbkJKbvAZ1V6B*R8&xgq;1RGW*_0Uj9GAml!E~
zTPnTI=K`i{4W}qAKd!e;D~CPk#}BNqRCjzN1`RQMWFj3EVC{^FTcw=jGYu`0&E51M@B*PG#ZUYCEYa3Wzzmu
z1&l0~p?zV5pb)f#yh)63m~sh{UV|&7ld&r5x&FswNqa$K$2|1Mk6?C14N1-Rd%iFq
z79R*C(m{}neB&O=?N8O#2@Rhf?(O|?aW1V}fjYZA6Bb5+n^5Hzb2hOA0R+zH0{0!6
z#ut5b)o+6M1MD#dzp%j7sz{~>B-E|Em(maU#z8W-LMG(Q)x+DdYD&H@ndVNcHpV*=VxXL
zEzK8+0?hl(>7?Z3JMyWho`Toy~Vj%kN;;rilvUV0*C?>e?WDQ@k&NUf9ki4QIQ&l{;
zZn?xzc9>-$AHTi38>3ddUnG?gZz~*2gDb7*-Vwrs+yDatT58NA^i^N97Wm)3^iWY!
zzRC{?3FP<+z@1`9XCJK`IG&#$Mt91H%bp*$4*~0O+M1hD*C~w79&lqWYU}0pB)tNSy~{qg1G~L>erD!VC(FNVk~gHYQ|&Rcad40SG1X&UzrKYnO#+5ObxbE#
z(DXo`{CumTtw~8tfrQ=Q)6{Ze0nMuh;hhFYFh8~zNPaWU#plp>Se`}+=?N$0rA0(M
zF6Clk3Q<$n%)V^VyE{t`1Tj}lO-;G6WE;KDzKu#;WIAuLEyE0tHp%m1kfh>OQyT~3
zeQ2A7jV%JLuAngZ5fdlI`s+nsPfs+3ia9Xk9dg^nKLJ!Lrwk6awEWS2+F0!lYFc{i
zflJr}pNbl*KSXKmyZ4{nt!le@;zm>1tPqhIBA#UAp7YNKv%TFM=RO^N{AOgP6O!PT
z=WL&qN*6$7n`}p2-T$k~#b8iHZ@+A8>IYeOkthMwvdht=o^g#%R3`bEV*YJByf}e>
zG22|9tx+Ih8)T!xNqxrE>5uSAE_Jka&Ggi1@UF%5U0O6dLdkGD9x*sM#@HrmzkSQq
zATUR0>@g%YDzp?l1`uK{ScQyB#%9U=DDAcH$@B#C=b+@|hhTasDq?cW27B2bqBt5_
zUTiCHR_>Op04WeO1pFmflNd2AZmayrd8AlYl#u|A-mWBeppbMFIURq4WSaTO8SK@t
z%P{!CS39EgE4i?((J>;MVj%!cYuu<0`brcqV~OYjeBH0Jv%$m(=)Nmf#$z>jE^0IjM`l`<8rbF
zeH@%O!#2k^eWV2(>ArbuKi{&huy<7Ms71vqXuc9Q%bS)OOUA^*aFioP7901(!3l?!
zx)K32B@XC?w~|ld*#HHeRktab!#D^n?T|u}KHSqmCz$V;#kIFS$>}x*+5*VJFK2Md
zgJ5XfX;1trkt5?}!`8vGde-$qd0W;2=7GH29~4D_;A0Z5db2qTj5Iec8IMd+V&1W+
zhbZ@dr;J+K++?i{rxXxK8qfb)R3xLb0;0Q{UpBo*fZ*ix^mJRb18{1RGBT&@C>kGW
zaC);h)pblHNcvUFrB^2Xd#kPOCffJ8QK4GO<&z#uy$Z=dX?G>lVA=;MVyqG2owKEa
zod8AhMcdh~4=8K<%0a$bd@#qFn8cAacMy&9CpIWoY&Bm@n7YpO9upQ03f3!n0H9KxMzi~zdm2A?aaQ?q(-!S@su{(qkww=|To}k7Z!q?{9$)E&
z$^lv^77GcHZf$=;$AyE1mICj)Y14%L-;_&-s1z&i#&9+c^YJ=LtR0FbJIS>_PAg0w
zvOzROY85J+jF$7oyhQbZPx3}-Uc&mi)i!m^@87>)Zq7s9ee~?zV@gU&X6E8m`7+3X
zuF9)y^Co&Kwiq@eIiaWbSl-rsXXH|hOX+*17{+hNgabOy^tBs@6)tq|71^Xoo<*eK
zdnQdz)yVE_(@RRthbwCQX%9x57+Ftu{onLX-|Br6)=MrSw~URWW0vyKB~8;ln#<>g
z&N8>cfh{Mu9TU*OP8vna3PUDNug6i+@;)YEMj#%5%LkER
z5;MT{>h3lG&RrKXU9qZHNxsh
zI7+l?YY)=BWR8+}v(3##)k>EFJ|P-SqRtDL*{8cq>2Bj+Jg(1o@WaC}lL<#MLlymx
zpk@ZKD25)Ok*PaI&YVmVPiP#E<V7LxsC%KaBC-3N?^N2(3A%AU
zeE87SZwZ(ySz7}3(m+rBeJH+e)v6QZE4OalLcAgX3k_2}(I?Oqd+1qNcIP>HCkuVD5XMMYrim>m&3~do>iKkC=+9L8k9!137st#rmapopR-2OV7-HB_1wCO
z-VNaWVp9)uvn`d7?|Ri9WYA&zxi|14a5LQ1{Eq^9&QGW
zdizDl%vc`)#)&D;1lrj!NEZ&>)z^L#D@Hd7dak-CNVrD^!3@mIxX{gLIY^(q)60G;
zmMCqLm6vA*=Ir?gw>wz-@gF|ZD-pN63(_J{;m-a(^eigVYrg&m^%@G7q!(S*_TGcr
zt9^1E&-nNPV>M56^060WEP-N2@9dnI(Kgl9sh`FUYlTosd3qu^ma_N&xx&KFonJoF
zMzU{_rbQt=N+Xt#WS6iTcmiuGrAQXXC0bWnES
z=1aOJA6=X5>DGY%H(%Cgisa$+g92f0pDA~7$zQPwwvXn47^2+Kak!LZYXeJ}=Yd%#
zUHPp>HqIXszxJ0*JgT@f4eQ^!a^(tI6V%l7K@S*eE)AM%Q)T+T9FGF?8KRP2HBGz`
zKFQ!GpB~_J-O?+kgpm~Pf1Rs(sx|-DTl=RuxlGE9m=a5ukbCv--!5WHMlPx5m2Q5>4Ado0vJo3XlD*War~c2opDO-yNtE=$}WSel~e|z-5=F
z2^u4d`D@cA)5q~Cx{G6cRtf(zDFvfwTpT}#hvSzjJdn1DnWVdmH2siATxZ;KKax&L
zN{U84wTNjIR>t6vT^ZPA5=f&Tt9+rVommaZ7iHDe2_{v0BdS(Vl
zD%CUdMZ5dPC{f{Yf{;9vfi?!f#cjlelIIIl#td_p3bNB;dn0d}>`_k6>Y2L3ZbwW~
z*H?Y}=+5x;(KcesLf~CeHw%wJ(^s};cY9Ca=T2X40YXD0ukX3ND<%mt_X>cXLxW-f
zxg3nf-YNf?@T4%8`?oKR8PMe*RHDL?{6T_8V{h@X|QPo~jjJ`FB_#KYdKLZ>L
zWc=xmN6zr=nZeuFu}cKa9tR~8Z1&$zAGdQq(PcJQ$Q7sN<-qA!Ql_n^fAi+eeJ1s<
z&-t8EE-q~iV4pfWUg!*82DAmM1;Hvldt+*s#N@P>4N+{$@X)#i66Ravn#w|OxO{9v
zf*ZIlpogJJv>&*01`W}VJ+;Xu&}sJW3+5`fn&fcf9IfLncaAXCadL222(JDSr6m)9
z2M(nP<7~e_fBr8ce|;79&05IOzZ`OY*ymOL{r9hAS1txe5k&IXy>3oodIvwSv;s#z
z{1#uVmsSj_gA3p_ST&=tbM8Q>|cuF+Caed)2n)^vw$FsaQa3H(k4Qs#d^
zO-BdOox=EV2xq=no#&WF!mS9t)3H#dlTlPo4aC`J8d7$H>$*ib2PYkRW;5Tob#D_9
zfyNW7h5R~2d?!cU{^T#x=%fn3J@C`d=KVOrgB998nLEohNobw?8p6CUF?^rtHyDnv
z{&*HI@fu@%eE@6|k38gWtE=B4AtB)x9Wi4zJQ1IRf7&`{>@ocPsV{nL_&LaH+CKhG
zSaiBH_G3%yr4(4C5-26jf^v@k2k!0AU`%1&tJ(IQR@j3Zcn>4Aka~@%oR0R6>hFM4
z7gdLU1Nec?5M7Hpfc*&~z1L}t8L?Gq2yr?|siN*{rU69p7B9Z{PgIfqY<%<$x3k>YQ`{?XI2wz
zZqk?U%J^NvCL2gcC@sbs8upJ{A_AAnP^yXG!@G10QP#uSuA(57U0|qTQ7a!n99na8zGC02H$0vl7}f|h`@jnsf7&E
zeR*lSu)=E0S4p`(>`X*VEZxv^hm^FNvFeSEGSz>6Q@o^sVkX)Eu-4;VN}%K1P>RC-
zd#>o~*KiTA?-x1#gV;CHGdQ!dl2f-R@$}Twe3|4@SYiC02OO5Yh&h}^i#@Pdy^v_j%c3B8Fs)m%oaQ3sq~R*Io|h0(3j!63u~Al{dPTRxNL
za0~caGhFmO;HzI;pK51(9}@@NIZ#!qW|sFUhMopwGwg4})<_IX3_pK<3H>Qu4bPO^w5nt-!D6-M-JvL4-5Vt(3Xlm@)_-1Irpn}%EB1`W
zAbUaJ^MFkUN$Yl-G;Ge=;gO&x9j~&L6%&L2SPuhLF2!(GtV~#{4^kaIyr)~PbML_e
zBt#qB%cB4A`pILr?7HLrWN;&PT{0qVU}+6!t_K17e`e>bne3m7kH#YTO~aBUqiE%q
zz!e?IWs^KDUT5HFhR=s=bgF@I4UYAPvem8!<&A3Np@@M2B`m$b;M{Hk`ze=>To{$j
zT{L!ZGL>y@`J&q_pT*}C)AjU+*-Qea4fu~Sv9P2XerzrEk%27`3bqe5s
z#>RpafIdr{1ZAm(*jMrO>rYh;x8Yyl>kq)XPm`WjG5uD@Xb%0+cX2;(V>CYlkxYE$
zz4frPeNS(1OgJ_S376$ON~?rloXL$yHMv%NdEyREqIQVd_MZ@a9H{oeEqSmzJbcbh
zc>g{c+o_9~jJ4xCh6%|2uw|3D{LZ95nVzQ=*2o}>_eEogHl=%~zQt14W~`07MS4jm
z2x{A;Rn)iJkAd2PF1Y^mhDzC70K|g78
z?}WX3Vqavua8_O2mrE)0w<)YX!25R+z)V&GRRHoGC8~J@`%7&mdOV5_U~kTvG48Wf
zbMXDBZr?Lm834;0==pi@5Tf>x0?r}u@_oaK_R+Z_K*KY^BiDoFn6z?D+uB{f!0@d%^nl{iv1^?E)mA72`+uQD*eW($L6
zKE%IgxgVG9?AxNn?;ozgyUoxaw6W>CyGaIW@eou+|6e`k~jCcEDF6qP(^hM)$@WJuvxi
zZ&KCQBSWJ76n@L{^qBVO%$0Pjd4Z-6>5&~2JEHZ$Pmzbjz-)!055
zRv6U0zyPD8KL_IlJqJ?uHP;CT9TnAUol>F!nw2;(Z%)5Pf!K_Y5V=Gs(m;T!Yihz8
z1zE#SL+}8?hcG@mVlURJ+*19#!W%M+aBb1d`F-l9a~-2bMjRO`+pz&wL|D3VbF25{
zDT0X*@#7W0%Ri={9PryVAjKzM1ZdLm9!p0ID8jD+Kx3bXDXLC!+H%`WLsa3J9|#Nl!d3JJ#1^B
zr|dRrUj}yVzCi+v+WG(3`|f|L`~QC>DH)~gP?;H#LbmKpi9*N@2RZhxWM^;6h^Xv!
zY$EH}D;zVCy&ZdfALsoUf5G>c@9p;e;kv!A_jPq0FR$|)kH>vIgl)vUj}vVAp*44n
z)K^khw;Re$9Or{mc6g=H#9@*&btIquiXAWk&qwym^1qi@bdZpbeN`5!dyDrC(A05o
zaq%(%ew`a%2%FSL3{BfC>PoP3U{g-@ZqG=L`QIOZS&OaOW9J&f{>XpZkFStq|U5mG`q$9@Z;qTho$m-bX6
z!6Z?qBL}s^%va}IOia=YcAEQ;T;TS7XO><9q^}EX`SRgF4X&`&$7W@>?k(>8=h^6g!mb45kZXqOppXVg|8fE92V4fK(aHOYDUa^2j}%2EP@fj+W-hdZf)Rh@3eVF#J3Kt?
z?2Or6FhP5#xwdw*$#8ITIygJeg&Olhxw-lD3c9Cfp7Q}UWI~V`ShPw>OaE)x+}@s{
zNJ_hBmn!Zx%MBSWmwAfGKr9x-J54f&z|l)A6l=^K01VsA?gv*
z^kq0h7H3{JPOqS7(rV6Sn8J&A@h(v>eyYYZaeBRn6;Hz(Np|zkZ#<1^FgZnnKVj=v
zvj~8kWjV`QA3(mgh2n*KkCLa1-K~nZp|9^gKR+1MQ;(02LEE8gn>diIzI}%TX9S(3
zE{4b$gJ&LaNC(v(~X1+HcsO&LaS7qC7sCcA3|c3~sv{AT5@g7CCEhdY?R
z4m!?tkdPSISg6Rq0k_n&EG-8+ds-dUSSA42dM?f;B)xP^SW`7vFtgVtJ+KQ6j9Gcb
z#e4!?{$F4P9qq2>OdR1hCNQ*f1;?MkM7v42Ao=uZKNMH{s3duxSCH4CafFX~59cx{
zilvGMu(g_ZM^%MTyyLQRX7Qw>X!XWF4Iuv8pgi|>RKXC<4VoJ9S76OGhdW@)4$OG3
zBwP+4WvUGuCF=(rWP$cOQQP#D>h(G>dCmWJ_BDQr11~Ou_H|lzh}W$BhFi`J=n-ieuv}puqk%5o$488V7c8glFigA&
zY;Fg?98KV|@eGmh7Y8xhCpWN|1*$v3mo1zRX)MQ3oBN#XP|@!|DWgPJ2rC_~V{W*1
zPNkqi1V|n#234P3tXj4NE^P!*$Q;N%DxHM;tq`T@oVD^`AENkq8kbwc2RuDhd{tLZ(WuF?L%cLGFBZXmfC}tK(oD!t%)wdM
zpFYd&QSpRXRP+;R%ZlF>$CvYD8t0sD><3RRg061DK)}*{GuhjmL(nBYEIZpGkA@Bx
z2?wq6--6!2!>&Q;SUXouJJg}N*2|YF`Zks(m2g98vWW-^#!K{Pfzj(G2p+M-ywXPt
zKu{hx2w$h7nxCJySi6#|Ti|ncD&Wrj+S1(}&ZX}Tb4|{xR!I(4UF!XNQxLfKN+OyO
z@2!*z+UA!($yS32B#nW~6->WdJR~g!-Ecwe`&x&98%`q)uv0s`bgq%m;;ND6U(X!?ZE8Yi};>(IRx5u)=_Wb<(z+eI2qqC#q
zEYBi+dk-S)!k2Q1KBa6Ab0GGBkhHT^$$~8oc~}sB1Mr!=oE%i!I{<27sT9_=4{Np5
z{HZ7?pg8swuj&f@(E}y(F(|HPdcK$dCp9;BIX4zqM5`{>Zo3o-o*LKtXV}GZUtu8F
zoFygK8hqif!W9h{kz5^UHxbg=GHU=tYw0iWu?H@dcpq^)bR`k{oH-g!L5`fJMYl?#
zFh6GC0lJIZz9|r50byr{f@-Ym{YhWy$US|~F)C7vKC;KYc1Oj~Jm{yqmsw_NJAw-^
zgr?Q3MUd|#UO?QsBhFExzC0)8K9lwK+_7f%LcCqB`}#e+7m;gGdgkWlpz{8g(FXYz
zi+f;JX9-tZ>D14;VMBoSKpnh?=rA?9Wb5k_GLaV!xao!dp}R
z+z7-Wp#AGp^{K@}@HsKozu5D8C7jnMy=L34DYaoe4!5_rwv0=@BrXkP_W-qybJ=+p
z^cd-amGWuVhk*)v6Sy<1XxcqMAoR`ADnf}LBE*2*gs+WO-1Cu`FUWvWQ>WJJ&(uVc
zoEu>OT5lC4#>dOMC5T5*alMdC{aH#`cgO+y=21EIVNG6A_`FM(a0EV-OAv3;Xqe0m
zIXq?6)gmWFzw3-1l*#DiJQS;C7C-F@p(uYg#^orU|Akpx#LJzZsvQsHDttUj{-^Am()?*s%LB#n%w;keIMO&&^njDH%X
zq9NDb**UA8UJSxJc;OLt!9n0@gEv2|xgL_nfd6$3xWfyCgqj67G8+!n(mgmJfuWtn
z<&3yf4|=7m5sUdinaLi^{%aGqUOha|;4S@X=dTDMsO1F(1=ZDY1T8q8Q~7wsenapX
z`%_>B1y@VvmY#_EoYn*%oq#a3kui-8(shd@v}nLIq<*(p2M@=8qV4$Y|H`
zX7FLuV=n
z73ViI_D4@j0E5%1lnXnpg$D7fa{SZkC(@vz7-(pCC-Kw__GiqoWqt0;N+jH!r6G&q
zGTw=U4N3|MY28?W70=`2FKTJLobB04S6BU^4du6Ffg>mgw_Y5SQLGN(yzs2OVoU9s
zqc??+%syhM&gN#b=3okJ|9%dh-ZJf$}UDu_hFnEb1|J8H$iA{n2
zT39Zi8gqrK4lYh`4AJ>03g9k1U_SIPzjUK3zx!5Txl1CGX5%n
z<>&A(Vj=4jfCbpbF3!$tBSlPQ(y)kd>6Vbo1~%3#fHr79+1o%%`RGW@xG4~>T#(xM
zXwz`Q
z;S_BDNe6TmuHEpVekNCm2%l``F{p8cMqEKb0c@r8Zsz6*pYqe1xy2Exx%9lc?Cn;!&!pI2&X3ii(tzFK;>gtSpLIg*!g&p
zP9sl|(N9v+*Zu$`rVTZi>*ilT#a$uK%H6%HvXTUh;{c(wKcYB%4xAckYmY9>Y82u-Scok!X%pr&|qOhB+p`oMeh{uF>c6I{R2kYm_
zZbl8n$fDEI(jc;78&189o9u>7vPGT{%Lvo%Kxl~rvF_J|NkHT&(8S2^(Op-o((;DF%hLPNUhiVbNBKyew6I|
z@A$#?5mLBru3g6cyxEQ%1S>}I>2SL<@khOv%>)I<_@p%Tddyw-Lr-5y%o-9-ZrH5!
z^^OOdJFAfSvgtT(KFbLFCrIvy7*H{oow&l`miW|$@t)Apo%NpTyH^UD)E=Oa)D21NRC&&Jtvuc7W=yp3fyD07tTMl3H58^*G2y&}Ypn#$a+IsAmo?
zyui42WXv_QJbYKa)8kU4d;w466Tz18*u$B&qlT|td7lwwNlN$PyiNAkzKqWf
zTsyH)gO9)V^xdpEt+O<>Q$?IsbZdN`<0J>yt0Z;XpAsEkpOp{7XWPWBK6N%FGuo8r
zpC?v#4#LADB2Re2^wEY)+Cx>5kNnv2w*T27b9?aweBNKyrCIIYFEdVzQ*kS6$i_m(
z@}}ll)vbM{n173ZIsyz*C)d7~H*epXa;_D%ia-ANJ=Vj5T}i
zJS}F>t)QqTD*hS~Jr3K55AjWpGOH%SJgB)9O{oa+(jM{m3mocO4TLgj
zD$D<+_f^hs9l-qND@@3PXSsU+30t|W`svk+=@N!39-7qMaU4=_>}R_Yjx40R!Y(GQ
zZHv_XR({^$)LNiMXFkTwwk)>d!&+Y#NI#H~R+sQWj{Gd_LXtsv-4S~;MsQtKiGo!S
z{}tfP5X=S00CdhngM)<=yboU*b5v9m_lRda5s;dC>c#WzgPLu|j<2${8R8lo883na
zD=RII_!3tMkF;R@BR=bPd_!AWwgxpz@(T*e%gDI8yN`~Ifdgl(v9YnanT3msOHi=<
z>sMKhSnX%eX6NU#v$9&6n!ww$e|T60ov4ry26A+DbwQnU0-{|gk!C3ax6UtKn
zA@T9?rKNuXzDP?$17Qcdz*69{WZiduBqm~@07TD4#_HfP@GV
zRzo-FBWTZ00StmEnyrq;Zz9F_`(cIP{6VvLmeMJDT``~>b`t{EX5e0FZReFtlQChz
z&wQ2ydYN@fbRCouzgZ1E>wc~OQ;kIkYCu$_g0ODs>ubrE>BLE1@BnXX_;
zaG6-Ds$DF~aS@*?;Uz(d9Q{eLku4~VnFr?y^4MJwdh_QkP$G48;)*|*3!(63KXyv8oBYw?#b_
zYy&R-S@BGeV=J$l41@)JOW=!|tzGqcqUmIXm=u@esH7oMC8yn(Q%+P~UVb;1anLU`
zl*FUv+zn>^ei^p%RGy|n3i$87Fz?O^7=I?MQ(-Hwt@B4Tx0%0w-IlWAOO$lqe*JnB4Wy0WP!~|#zqYi5IF^q$cwQ9#0fh*NKm;yRz(QgGL=@Kp0GodA
zd2D2Xblsrfl&5sv1SF$m0c!4bhy^KNlpQWk{TLFCA?=OLGI%8vz969LJb(YtS}CzU
zZ^{@YR_U^C0E>!=DFgB{2Rz0G2eE+knv`dZL8G@0igX(r?s2q%fx$`;TsIIo29v%)3P}Z`H
zX=`a|>FJG)j3@{soRBJQfqnx3@M8HKIy69GVa#{>8KbzUuKS8$#G6n4^arv18lF_^
zY{JrQ#R!)6qr2EDga31eF)|BoMk#c#N%YP}k8zECUH!a@=(_}Q&76mwDUZ94@^W((
zY>fimB}9A!FV8QA8r$*QSS%aJG1%;5^e1^NqG!)&HmnxltZXIQq?t
z!_M5iMf)uW5McVyhh)F6KE<`|5&jmER0m|+>e6V+7r-IJbSU++}9ULkGC}_SUym$
zWa|^rt_`2|){0yfSslM1$tE~ZNC&GiUL)n{h(@EgurW0q-2mGr;HU_KZ&zUL9lqEm
zCfVdWJO`CVAA0qJGJO1hKaWl5va@hyb1Yj41@HT^($bF*$ROd%2xdhOkt|w-Uwi)W^YdefGBSZ`Lr)3_M$Cs}E
zZS}*Q88NBjDOvTf%MRW+pw7z)Jb(WDkoXi*LgCk;_wn&LOml$rFpTikOP$}+qmvIf
zYaU38PsLcd%bUdumf#MpI~m8bSx2LRYPLo!31BH4jjOVtVbI}sLj+Wn6c>L{pUTV2
zgU6C@^k7YeSUt*jZw;*qB#~o`XjSnC^-}Kg1ZG3U|MN+5^#zEmRdN)ad#B6(=P~7K
zuka!-t1h@w#l7~hSW`|(`sS9F{?^ZhrKP4Jl0QYh2a_^uziUPiIO1x}$tR2CqN^A2
z@QkxT(WV_MYxqWFf9Zzaxe2L|j756>FLQ}CZ>G*D3Ss5X^
z>Kdt`NEqeu*!ZBl@lX{FjR#|oJR6lVM%-+~lgDrJiBng?-X%Di)X-?|ibqkT%z`Fs
zB^13~!qfL_GwvwOj1cX}b`LKO?7=~VIzJc_8F|fX0kqZd*>6Jg((r>+74GV#Ge0+^
zj95KbQh1Gcj6ar53{A|I$D%~QMgdHG`i95CR{s9bSU7^(@1+^+TFe7%80{1F>if}J
zGIDao6u-WG{pyDZmlq6@X>|6f0aOW2?4{-9?99wUuN(c!RthZNEvl@nt-U-wZTtGSRB6(I*an
z_#Xm}3||I&u5Eg>o@L(0JPq<9mhvKb>4RGAwjVW$4o)A>m`~$@i74fLl`sFQd{-Hi
zm{VdEInQ7Dl}o;IwkyQfW3-xibcNMX-JV8s_w+acV(K&2xA1F`h6T
zolrVIm+y`K1-rQlb~79ZGY`;zvqur-cN}Y1Ic&JL+jbpP97HHx!#+GzZFL$zc+U)T
zBKEG+R}@)D7dWJDphKGk?%6~c8k!zO`kj?z-Ilh4YKZGy0Cszl+MA
z!8WGTrW7l>6th=+uX@WauQM0@rvZUu12whIoT^A#E8gYIdFe8u(UrBlt60Bk)GS%+
zI3ujv#e$f*&^~n=y5_w73Dtn3H~l9B@priP^*93pN7nAro8uyFdnST#X_FMc6r7^z
zLOWhHa=M6Fzg9$L4wZ|1bUr*x|JHyqsa5!s0tbe
zXdS+n)n-efZJ>Z&C#&&q0qMest;Uc1d>5)cW~V%S%a*lDW+pE;<|e_fx+9$Bk$x^D
z7%>vR-`rHSTjk@*MZ{bTRi1DNrAfA{+kT2^C*Ec9yYn3}+>vqdoCPDR#APnxXoU_%
zg_%?%rlL^#-e01-?|3V(b**v-ob;$vNu(~HOsLA=$xFL(BQQ5a&6e%`yT3LqL2$tAMGyrpU2;3H@W4
z)7tBee^it%CW9Bed0Z`x8*)Od{XEp(`(J%47xqSfka5em;lGP|)F$K=tZL|R$Q
ziv3%4+kz`{L*olL$tZkT_#%i8Fe;zWzm5WP(|Jh=M*ipXWb~yxc4F1#+
zenI*s#v&D6`{5fGumLmW5Mnx=PV;0dN0;(t*og|yzM5pw_{NYAlIfjIz1PLNBd-#l
zX-~D|PC2}&u~$)JV{04E;^Ut_lv8y3PzVxF)7cH=lsFmEb-CC;D6bM~1
z;*3eJt20nm4w>ZbhGp>J!2`@vPxTD5yMZ?oQ|0$Fu)>FV^EF$^1LiuS8a6`ouzPI0SGwQ6ES?Apq9rqp?*`U9n=*h`RtoD6S
zGq`|R7C5`ADq-i$!@oR9_S0S5VYMDLxF?cWPRvNoTKrex
z1=AZfczCa&Na?iP1WCupm&bn!(n;{|jrF=HQ-anv-u8owGAZ-Sj$fwIsmMhFg06+o
z`SMer%}W9YkC#dtZvR$&aSOo`YPLf8Gy6wDfkn6f*$
z`2lycNXW?`XA1I2)R*Up7(8d)X4|PB4+?;!NA`EV->#3E8qhy3PX-m#NhN3}
zR9H2Yke~)K=wWb|uUt2xy5l1b2^DDV#N^}-udPrJxd=n@x%0K)IBkx9@2^^(iQjnE
z$2+bQSnt1EKV#IOb0!*hSH;$;YZYpf_E0x?+qXW9Rz1qm^iDc>$tBcB`b{9M*nop36CRX%eSAk~Q
zVGc3C&eDRa`4MMjNq2M?PI30T5|#gcQy2snj?BA=7V>^v)r@Eu=RUZFwnR-JpGteP
z)KgJj8dIo%G_w$@am#O!kzjDTIhOYb#^%)acuZMZYmO#TDq0_#gY@Lx>M1q6Js9&b
z6K`%|feQ=)1A*gCXL~D~;FgdVK`!I5Y>9Ui9izCPUCZ)+{ZeCN+tq;;F6#_>x5b&s
z0PVrWf@9+z%yol}?r;ZUJ)Jkw#Fjpfe=<20dhshOqeS^pN8tFZ+@QjkRZQgP%;GRW
zqbq71NA8B;%GRrlP}r|d1{~yQ6>J6UsG8v?re|h-wFO)=b1uGBS6XJh^8Q(J1DJ-
z((~}BPDx>Sz70CIti9{TBW%P6?9k|9_xJ<_EjSOI|Fy>grF*vVi)`F6t@P}4uO1ri
zWKp**=-DNJ=7ZBHD&u=zp6Zu%fEb*doWOnyqtb9{Dmd!MsT9eXTK@s?1n(G1MF>|9
z4Gmp6ya3U*@Hh34F=xIiT5QBD!RbtNJs^-zxKc=t6ipVTaCnF2qd{M9nVI=!Xfdht
zm5ZUKy51DDRgaw#XaGr)QHQdyNK0wT!u9qslUm0`JGfvF(^d06(MeFQ8Mn{YSeBP%
ztQt4joW1WgN^mQ#`OcRukb2_idf9}|2EJBjXJ<1rA`u){*LAQ>MCYci37LA*!$aZN+F=2c^NQbT^F6aD!Bxvya1I*BP6{>eg@)3J{V2~pcNRIqD-p5nhQQ-fY1QI!2y8)
z0zwe?B>aWbGm~a5EG%5|#b(^^#FvD8-eBd%xvAYs?Ul(bgUUoM@37hQ^)Kd&mpI
zaA2ShpoQe-=B}KTM$%qJjQ_q(J`FQqC{91o>9A6fUe?poLmfpMCxDW#TynZLo@g2d
zB5t7y$4J`pb6{4Ry_e3VhssTV=asiX2#cDAmNse{Bxv5-b0n$b3RIV$
zjdS3VxkhH?jK^RCVg&&%XK>9c<%F#qhBbm5CQq+C%qJJp6I92~^H-$~m=Lm`Ka*jZ
zhxB3u!PlCa@t&S_B{KIkA{tXyhmf2!e6(-GQ
z8U(GdyoT?*=m`?((6jp$A6@csPko=5%m~TU7xLkL&V{2qgDTEXXHg1MV&irUM{hZ+
z$=w1NQVb1CdoA&YWxb_Oqt~>!inXv|#lb#J+BIHuw6s6IEfd2$n`SUNh(W1`)9fTZ
z2>7(g1JEnj$#o|bCUJx0K&`IW08fo+*zyG52+R`TMB_sZ#@q%ngM*`^Xc8DZ*@i|(
zDljvjIXbpg-d0*8W;r-esxXT|+(uiDUyRsyhe|SZEnuLH%1Bv)SxoG6HA=3ZfHAJZ
zwM8tz`DoXsSa5&V+}QZOTu~4u*z)YdaRc#uDCn$6r9cOqAa4wZ*K+M=HUs?
z)4pJ5XXh>pq5?BccE+7BewM=?pVMF7GME~-2S?4~{vMtz^a^VPcjsGgenN%ai>7ir
zbJ-xVoN}z^czZl25%9jvPJ3KMqrHi_CLsMQHVhe>_X-C#$;hG6xc;49!Qa+hv2uSF
zb$XC*)+3-^au7P1;6`d)IXf3k$GdpyxrJshls+{=3G4h~T&u2Z$2fN=7B7{(mQ^n3
z&MIH6sUbX1K`|z-b2guWUj(La^7#cGocD~<^0leiU86gXqIxS7WpGoa+S`@ZoaIB;
zmhHH?8SkgNfnElR>NZKfPh1}qZ7-k^-Pq{AWOr}-k6^o_N9)&}N`n`G~md**$nKL@*FT
zj{Fk6!qXf+3{msq$`Z3;Cy&I!%Mn%bsyXcVzmexN(RxyD?SOR7Mwi)&(CJR8vdbHf
zsWE!xBbI#q@)IXl+Hd65e2Qfp5Ig(&F)9pmqcHZ~R8nmTiKuq$)y2?=#%OQiO#6SJtQssg=1`)Yxo
zkMEAMUg4{Am^19WsYJ{kYczV{yq`(b7$i825C#8u>2UpyPjK54IB>l@k6-y*#B!9$
z5u-z8_{cjeY;G0hKlL3!+`KPejj-G9FZ9JWQu
ztkosGRlgB(tz05uphB)4<-V9Ct0%Ac_aI=1Bl!oFL?3Id#Arc#9(M$ae|0sb#8k=!
zQSz~z&R>V+vb%_)t!edCof8sC3-gmbl)hOd%*K8G+@;OVRJj?Rs`}~CeHej!)Z5V!
zQXSXda}ofOpD++x*@qw<96Xc>gp80^V5Z3y^#RF=5L)>`m7UiOH6p3^ltFr#&(|(6
z%qEUbPG^V9JZp0Xlb{>LrGb+Ufe{oWvl5U*
z#ye@g^KLj&Y{ml_UGP8Ulgprv@I?u
zUK43pa6m~kbtW)qz#O=Mg5l_I-)SI`V+(Zwq|Au&_v7l0ugGZ!K1$`YKF=yI?WlL}
zoy64rI+~zBDeldSWQolepd!zhY3y8$=^FI+W=3hBA2?9AnHKkw8Tpr%mkU08`0Uv;
zFi&P?XSc$vI6pto<40=l!WiPM+d_2tIXN@;@(QyPrGLV}qe;=?bBN}`DQ?9txC`nB
z=(gJG?%`PZV-!)yg-?AV0w4s@djUGR2WD}hB!Yv3rgwaYJ>SxyCD>v5|nStD`8vL2t*bm2job
z1>%JO9#~sSu*Fcv7<2K^Z|YSjcqL_{!mNJv=4Nj}HLQ34om`xQ6Z6>B;#
z{x6$unh5mgrwto&L_iDeUvY$|YFxri{E#31upfXC-}kpIzIo_0_9I~&QiqiaUI$K!
zZCu7RrZ_$5=`obR%n}l*)palwG9D&u-nb!i`tlr$F06k%MVZIXzJYSB7ho`-fOlbG
zP~y3KD3E`Cd53#7;Pa%R)zNcC^#W)M?K~u|$&Xf;q?1OP{d%))DsMsMqRckD#gUA`
zfdG+ksET?$?>bue??oFy*;TqA4V5QSOLfQjm&HZnv_sf+?kq
zzausv>y^1Ym#1+y>!+t|oOE1EK~7!WHr^CeSX2v8x3%hWZiFD5YIfXM22H_Q>P^gj;rtD!*HG&hq22!K|
zq;4Z0q0p#CZA9HJUO>361DFhy0`AZdTTR0
z=Y5{8Z20%i+HPAaPs@gVh)jV4gWI?#v(yQ*AgNag%P>xy(kWE^&0N8o9acpeQYLad
z=Ia@(*aV-)3p}K{*w}mh-fp?Ede7T#2%?mR#mrjI_WHA1;u80&@%}2M4?y!W7}dry3fs%*=KgLLN>-)CcktkYEVn1L_90Z;q@{HCS
z2E>cdr;YlrqOE{{z_9^KO>Mw2$xTKGC6l%52VVe$UMTmA=w%N*RsI^1-(tURH8YAB
z_?G$ibgt<571CqOYjZWp?zV)@<>9*f#L8Stc4&A3Oc$b(k7u<Rvb!|MkH7EjNE
z?B`e|fz`;o(@Zd^)HCImZhzT+kh7U4|HN1J;;dRVp{>!C^&!TXTejyBF5Cg03
z7^A1+9sq4NHmmS|D*aL+>7lGltN!TU=!Jt%*kb&Pk44d4qoY+8;dL{B(rL5%Mn?8Q
zKc(pEj6~k?S;4GbyCqzZ;*q>tUu8jg+dij2*{0?lvtU?e{!W@StIPH*oR+rs_=i_;
z;O=h3I5ZkkeP!%9mICXkR>>Q-cUqnLqFlmrYud?D4qx3qFhxylu`3o(%|*SZA)|`U
z?_wZ4bgzC(3rrxmcK6|qUyMeigT*Hmma%|99Q{-Qpr?EtMw~Y?OG_QW2#O7cHGR;K
zV*?%<>bc9m!0-igop+#6e$
zSTRDKkD1kR6bjNB)kZhtS0~&5L5E6jCFh%DrD)Fac4`BJm_{BkeyV88Tmo94eISR$
zOCpcWDQzYo(!R91EJL-m_3g6mLg{VL@)*XLB+>01V0xF21FU%X`G5EKBSA}mtt@*9
zk+yG~oXqAzZ>9m;)gN;BH-u*-_^~wtQ3Eqbx7#mVzaFg)n&ZEWXO!nA;|;wn5r~(N
zgQB7h-5H!(@TG$delx_lk@>QnoSdBJdK8PYf*6e~_e`G5oy~m($lw`Ts6)6e4{@Uk
z41vr&X&O|21w6Bl2;@nH=i#y3e#cS&Lw19$N}VwoCoX<_9~n)DHW}sF1v*@bTw=^t
zXb+uvv|aA8NFh*Yw~x;6IzoH%Ki{rS4wE@N6-WV@6a=|co~xDr#zroMv!Wix=-z|%
z1ce6K79!b{!#xpz78A~k?@R@vNEC{{Jl_O$%F)5WWz8YH$Qlc<8+$rUf*!`f!6DHY
zjQKJJfSbaW+~UFOUFd#dTGYs^h}wp30J{XI*cj-h!2)4F5FNp?tuGy#2;+KHR8<%M
z`3$o3URyR(+WEV@X%g;lTbrtj8Clk1SNbpu02HqR6H8@t42$ii=(nTMHm=c891DnF
zpoR=DQZ3QB@$x1*w$j{P-K+4;*X^OU{zuanqHgB7s%LD)qcU8Pha}HCtWL$IDHbH0t2J0M0B!sv>s`W(C&FYv__NcFK@|M<12>kkw
zwDeU3kYCLGL_yCYv)z$E**&Bmwzm(lgj(S@dNNh9)CEN~6pxL?Jjxut*9>fg2NFkZP|6$V>W)H0D})GNyK$q*D+)YI5P}1U1w@GKJ}6{_gONE5t^l9WH%~A{
zCWJl<6bz%kpJwFxs@W#z{(HGd(A`5dS^v?t|N9#h*?%a~#PKMO{)Ji17d_(7vR*`f
zwIBD2^1Q-DOxGzI!Av5YS(WFE*$wd>diR{o!D<^rp^?*-+6>PqLTnT{>^YXF=8$!Y
zO^>EKP)D7=K+f4e67H|>p0n303>^KBjbgUVQeIM9+Pq}|m!cSE&x=eX{06)gw+EGU)2B)nYUY^;9l
z1tki&U%hknf*}0;KoZn}LWUacd3b-@kuj)5AZ01nI`DM>%n%
z%n7a;JgD^ADBSP)zrY?SpBE*cPCi3xOO}t1o_Cl3e7Uwo`&+@cZ|sNI+Wh>mJUsq?
z3kwR2hl9^lP@6{oNb($SxkW$e(3TV&9x>>t#P})x_hTWE)i;bD6Grv^eT_O_1zi_0sD=6}a!P2lrgjL|Pp#U>O$5@$t3I_NImbr_R~_wqxZY1S9rRSAItH7|Z&3
z6x?8vrXUDNvDk6{n(Tq%0U?Qj=~16ZJHNvPv5kzJoQ~9LC=-%aFXHa2Edc*D8&osC
zoZl-Ud8XrKEvIw+=&lCT_KUMw^D$_glqm3HRCKb8t{Pz!9G>ueVm+Dgk7^0n>
z<%0UH`L)BtL--Ev0^-F{R?8Htgt_xp{r6BGa8nXo(f`{dFsU?lRy;&Cu=Jx>opVX_(z8G}vgqIfvLBGIJdu@ch79KgX$g>Pp)S3;npsQ=t7fX1A3o!%
z$P;G4{j}U9QQ^wZSE|w4-@XqGX&++6I5@mpL)b%)#^wXptTSfY9Z_VHbxvsPm*}xTcdy1^3eNojmzS5qO3Dj2xcqqW2&9oeoqgTG$DY15>H+p_=bLzfM&s@kQP1&66^3z(;x{oxY!xtt=yeS0oG_g%|lcKam
zb6zQ@+eNF(Z=hF2GA`ldD|pYM*`w$l*#D&+UIm$&ymKE&=EqMDKs^CR541a2#bYpL
zN5=W<{27b}y{uQR5ZtzIPEP!KL|y|`pA{q;C=&I0{v&!L0?`{&ocM%H08Www55Ewq<)wO>8dVk&LOgXh3W(%!>auP02#NBUvI1w59_Qso;R%9T1Ms#vR
zG#73w;XR9Z5tZz=f)6N2vxG6{s|!H57^~0oKLozr6oI(O$k?OpOQfWx#sHIkJw8?g
zep>juw?IxOz~A<2>UawS$?C$2aR_{pJvq%Lt2*SDDc8QF;igCxUz!
z0Kmx_w}J8T&aewwFt!l4fOjjL$6!!{9tnQo>SGQsOxH0#Ya&QNG~A38ZJVO9efp=n
z?voVQz25^d6wA5JQ2
zIUuo8n-Qs6x9F
zoVy3LC+OFi=%fXL5W8^oKu&^eT{EbW8Nt-NF)aL+8X(wO=wFjS1){4DCC2GlkOw
zSdvC06F+XHyQFqOe;mXkFB7|WL-C496BQF^yr%8fGAN3}v{ANA+dDh@ARhG?BI8h#
zmnYQhVE=>;5v3n1&HO9Q^QU7}1iTKN0f}a+20>cP{;S2xdkWu((LXxat?$ID(9rBx
z%aVT%sk?7#=a7d$D91nan;&#z(-M-6)vLc-d06}>%B4nnO^M}j&
z;Gh7ezMwifb2-8C`qRUx|8+=tCNmsOFxFy+39e^wS`I9aQIdSql`avciT
z?wkEET(YUB2j<=X!eB6ylhCdmfdB;-^b9M3kjHLrQ4ub*8zh0@H-)}rXS=|NOV|Y9
z1`oLK5p2n4&{u&Mc@&@{*au}*|823jm3j>K_xLO$^d>kG0q`SLIs0v&2)_u~NE>l?
zEV~Y$AVDDX^&o!-XuZw7Qb!Wpu5B&j)h<0y)AjZB0bGq>L+3#3-pcTTG#GmU$%a1)
zF-}raQjmKKq$2=u(1L1fQ(~MdD=Wbf4*)j^)`2L8q&FC54|;T{$bo`{=YRUW6y!3j
zCOwN1k6m5&!N7q|H!9H;L>AsNtozuRb%Q55C$V+<&@Kb}zBTpI(IoTIaE?85N$8Np
zFAB@2I)pmN(9*6yibGTvkwp6f{~Rw=KNchYT;jK*!qgX*>&GBl4CbxUg0K&hUcOZL
z=RxSKS&LJ2%aC_x+?Y@M5?E6uh0-IS7^I@}dr5IoApbK7zX>7UH!kwK6XW9wpIG~1
zj+a(f^)=Gkl9&IOoERPSti}*o0_$JX0%xVP=6`l4*Ncx<
zbwBARWEP3wR7T$(QYhQ@@QHrF-~0_7@<&i2ykqeIv%
z-eaX%dSH|8Y9X@-Z_%+H%XwB`wGGo_IrnOTi$8>_C0*Ut!feu0S}0+l)drl`qIy&8E(xJ4g8%{l&AXYZ!2W9|r!08X
z7eTh-XTNZL)J5*v6Svy@6d{?I%N+@*j{N%hM3XDDorV;j9lJ1Rk9ra^8}F2`#(zJDclC1ppp|fmswmAVD}D(-{PM(v@XO06
zS9jLmY63$fREI~$YPlY#<>Lp`|4;yh@9d1cFtz+tXfU#(sXr1q(P_eXDI{@d_$AXc
z%Y9fX$6LNQtLf%68$ZS_wXe6)zzWXIx3|B*a5x^b{@~2uQg%g8L+)yUaQXqd?K}LH
z+<6=!^g~@N1IjN%(~oQrAcj1gy*P-b9k~w$8l*wGW*z^Ti{TR3=j{FNHhf_J&E*u}WT&f#v!+;0spXN6n&
zG-@>&<3VMDu68+
zAi|997M%2F3Na;CGhu;?UfzPQY5Xn^fVvg=6ov|DNqY!t9T5Tg`)yr_mhW_*;Y8wR3LX(n?xx5R_hzw0mHkhb_L+b<-SPfM#z}}
z9RcM#3i3=$*=6!aa60kaKx=|=a~J*h
zuWV{+=Crr`c*d}k^)IgH_82OVSIDBm?YJvb<(VlN;q}#K1(f~JPaTIwC5jaBzt389
z{k;CN?vtj;Il%BCNmKbp8My_I%qx*sbI>PaYCWYBFFi5d_HUd-k#{PE%3(6-@x!#q
zXLofk!UOtH*T|Au1SD1b;K=NXl|�l(*ELB8ayZ}HgIRV%FPvi6?@|3O$#
zaxjr!K`bja
z`LSu&N!q9c#}=xbxPK#T;|*`+nKYoLm>QaZ7~Gu}g5)g!dtSSL*T_r_o9&bCbiFXq
zTbzW%7I_|{Wc?d<3x_A_tfKUa1miy|u_rRiraoDah2=WlEc!a#-;lC>ougp`JVpo>O@jFbA|Bk#XRysyp$WS(mL@@fY
zwa?+#pVHnTCjrlkC$c
zEqE9Pn6<~1PH4i7p#1MIfe=P%e!!`4|NGdM41U{qAGxlAn^|Q^yA7RVwI925vvbtc
z{`mJ~49xIlpLDWKOsoQ_c+O~?V26SDx)rx0kvlCV`4HoqI(CJ`n+FBgWL+w&H{h_}d!Sn#_@h)JH9A+n%bcuegzH~dH$d5v>#EvZ)6&9i2YG8K1V%9O
zNfiM0&z!M@2n1(#yQA{@={-Gf2`SSYe`(iYk#
zB$C+{ur1&VvediSFw0(x$h(MfaA3ke@jAmJ@J75}0huMavR|)#61kV0!m=VOK2#Jy
zi-96+lp^l#oMxEbWd>RKe|`DzeeSQ2>$Di~!VSM88us;?V2RJCGY!JYSEOByJJKN^
zO9|5;Lp%29rFf{!7m*9Nh*u8Qs=NJ2Wtn-&hX;Zz89z>OOI{%Fei#Gp{`=TQ*<;NQ
z+LHv|g4b#t{|&vnBYcnh{OI5)VWgV=RrOlodA4#T&Q5)ZTj7iO5Gg8uBUPew{Mrk_
zCc@)sUoUxE0{A%hN#s?}#|9=MRB6hwBA!@xT%5k^Zot0H$e34%Ix_jT`pWL&a%%j_
z>g8ILA0v!^P!m};`Cu<9eKV2_;84?ry#7n
zE0Gs@$b2G$nc@EYy=4?bCfZbwg21LQ{1yeQ*lv6mu(U|k<+U)%HmcR9DM5%$^po{Mzs&Y|opalivmQ5c)*E}_WG
z+E48(ajU^Q{_(rc=#FH`IOEJDM-4?hdL}h&$Ir$3ZMtjmWgkYWje!Z+za@oo1Ev=_
zbwP;XoJ2-xPj_jENr<$k&IpJ<=%qX!qDTz<EsK^3va_KtT(4))Tog?}Doj
zN;nl+?olQNqd59t?USxis)*08gf&$xxf1?OcobhanUa`c+NK^(Ox+VH#fm6O_Z@w?
zaknqK246O1Bf4sm^CL_zS2+xoVF~Huc`Cs&Gs=|(E#zU>CV*?~3!Rf6ur@4$1zJ3L_;a)p5Vu%YyUX+Uiuo(T4Djs>J}_
z;=n1E3s>JAlvhHpHed~l{80wIjn|zqp}s_jxxJ3*vq;tp7E}}|Hh4b%HC@s2qnSsV
zErb17)n_H{`P)-0bMNNgdE}=gM-Zc-yh<;q;wpbHI*30t5X=950(&*Z^ZIW8mg{FK
zjy$0c0taYOaj@rHS4gnACK%%?UXUigE|=|uP{}>|S=F!aql0u|!qkJgrHfr*evi*M
zyH5Xe(PbSRRU#8uMhcd6WDic@JLGobv=MkXYu?vdi|e6FZ#S$|owr;==1@;%Z%NTY
z(A3MgJm!LN825T?MkGf2@nx0k-Rr0`rJyn(_?2xg5UY0GHKR1P-4^(i+=4DEQUY_)-&sDJuC
zvQ1=aSYOaa$$L|vLW4K?f*lR<wQa642e>_yEQjBCdXxFztLB!x}tH`z=2K;~&Rl|t>SDD^O;m!F7bS;avMN(u;rm&3V&@7-k2Ml!`R1EbD%ry0Mgr&;vuVlJmcj#Xjqdqu*z0r_!&S;LX0p{MNs!RF9)apE-T
z$!{q+b8|I_kbi}Hb%JNIe&7
z9iA;9|B-WI;8&pf1a5X9OY(qbBD3i*ean64_#yEh9T#rb-GfDzrjf%rr2YX_da|7E
zwiR_2J*%@-!YLlpQ@RXefLnUb$&QpHtGdY`LDej$}03a@8yu^{MzT^SMO$ovL-NJ^Vx(~U@k&_tf
zljkqv|7*gE6j9ku2d#pNIqq-&<-oQMgfp1L=hZ`@IME+VF9g4S4!C84LDyaa!X)m=
znU4|q>1r!cB=2uFZzgOgAhkdIX6q$#gmDr)F=7)M7&%H*6zsrZ7g@gfX^{$CH7M-^)@dIKNpzrb0LZr(5Q_v13W%q%g2Evh
z4uvyFvT3*+Mzo=TTbdr=$dG*eTl(G6AnUjZy-EvEkkdf`yk8Ccx}XdNFqb}%joSpH
zaSe@YK;%+TP(a=3MeH#EE`Vh#`2KxbMn-7ivxp;y2|`&}ezEp!b)J^fOh)>eM9a7D
z6qydh$Rv3ftr9G%zby&V
z9U`@SjY{)XEQXfmEUN3Za*BvDs6v23k$P33XKPbMk60A(1a10je#MKAAo_!0o2<3)
z<3RLhkE&DfQp+RkCEovV1sGE80ipVElxa(#f`lV4HZN7qtTudVl034^~@f?2s^XCepsYPolQ{PW9R8A#n6h*WYztwG*;FPxlcB*os%%To8Qa
z^FA~m0
z6nRMD69e^V0FkjWFg)Y?3&l9H6M&#Vffx)lKsFYs)v!6%?_Nev8W^HWmVkDBW@TDo
z_()B4Z2Ib@xYmlo+7p?l@d`Yc?56I*kxm54jtGv#GB{kFo~D}@i7PwPnpInnMQy+l
zr5BOgcWVj-<<+Dnd^YPGvR=zL<9sly>Qig=dTr3PAq)hBjH7
zW+>FJ>N9kl)$TjLm^`yg^Aw4mfUb$qEeW6*Ztby!K%Hk8hB`TW?WFuX$!
zoU)geT3TDd)`yIeGHfnPxUJP6*h&vMK}rMKq{0kCBO~MwbU`KsgjlqgurRxU+AZ*P
z4ulW!K~e;I5>QA*stpbg4+FCm%KB4N9Is!$2CWa^q?eV2>okD!c9@Q^^a7xu5TExq
zpd(gfxh(qH^O0467}x%V1M%X>Gwqfy1+vBTzuPv7U2=jSpQ)ALc49
zaaYoAH0xN?%G#v*KK(+TiBilyCYi}NnP5k~QomEAuKS3w)$fDy3sX_|PI#0lC)s=|=33RDV--!ap
znDDpNbyK7&&8&kE=&dpWOY$h3WcYhk6}!_gUqWO>uM*e6rX*V+W*?zXSc@|HHUSA5
z+xMF&n3V8+3p)MeS2RlP?a!1$JLDU8_YqkA)~XF{Hcs*q08hX9AGG5U0h{O*S}5x2
z@KjyUfo
zNqol{>JL{Baw98>1t2}heh>6{m11nV_tT<C1=o(E==(5IJ_l(e(6gRW;u8xWR3cZS>b
zV!XGvx2ahVu93$HN|$tpKhsy
zfAYzCAP`LYU0U|P|8A!p&pPeYDa?h%Gq6vW=7rDbMqRoVSS^blO*7SSd|Pq;@xuzN
z6HSeb9HAQy(z-v^o@$y!Lvs(#!Po>=9}U;_zQR|$sJam>f&Is;%~OV}n99)hyiBh`
zU`31Zv3bV1=LUZ3<)8#Y_K-=GjkS?ki8K|S$69~F0kEt=co^JvIf1ERh8FXRn|pR(
zV6XcJ6P%Kn`m!sq8CVmcIH;ynbL}4^irw*u>+hnWB&iCU#cUY*=0@Y_aVjfKrBwBC
ztlzD}cRJl&37axiQzQQ6bz&_wr`s7$1sF{0Tm7uiIf0%CoVSeZ4bZ|bd&HZNXx6qn
zzxAeFxP&74C>FCnjz_a
zJ!qyZsa~)0T#HDK9ZroUvvufJvb%i>g?iXBujP>HFV7I?51(j7+gBFGn2FfKu2-k47geJ}L#YBN`~KV`rcX2c
z2L+X(PT&H#gn<>FMOOU!O0*
zd|2{dU0UiVosk(Dww<7LTJIsywtIqgdbPV&QK2I5fCDfc_xhplOGh+4iu*+^LgX=Q+6WAymjY?@7ZWg{ppJ9?J`J=&y070pssr-QQSXhBGzWaiPN@IWJ5?8(g?xDQlwHub9*QTADe+^uT$v7SDyD`31&&SqH?ktO${@B
zbPWUO{UQ(+E_(04dM>-^l>%_Rt6CNp1Ra)vD=ZB1YNG**Yt62Lws_A3fYaTwD7mi*p!~Prv>K
z;|<*QZO}zenqja88E!uevu1F}N0ly^8FcWcD>o|pL
z_yq6DcEzz$hG!0PFB=+|E=jf<;A+g>qJ0gB2aC;mg8|Y>EdDXzB2@imH393^vUgF=
z7O#cH&BExt29TB0h=K1*6~BEFqH!{&=rU*e$!jQr>eEcrCtRIPrWOr(sd*Iwp0GrZRNDud3KB6CNDU4wk=6
z$B}UcwY4BDgv4ovVjaoH%9D1=*9W}6#TV~RLOJL%;R-V1pl)U1v^6vij$@bqiuDnJCC%GP(&ef|usR??_;tS6qfajrD
zZiJiPPk3c0=WJQC79&8-OTkbV!z1;JaHI_S7H1z`giZ}TQsAcLlsrzueRe=P{YAF=pEsgvYoUOVh(j~<6ie?@OPwJ1{Zr_D$C0dx1{7Ora2l8RJ9y2I{yrG#|~
z?lbL2p96TL-dgr8fjsr;ypxCsTE0|jMg|3(ISN0(?PGGr4mg=K!D|~xm(PCgdmFke
zd7&d=1;A?pi3(kprvU)!Z%lNgrlwX-W(wqySiR+=0naT}b8_LB`F${^jdhYS>ua@!
z6~|T_A21@5c9==4S|cg;7Juhz>Xm5~dEyDOr&kxtODTw-4;z$s*L({9UN?qA}>N!1m#$H
z?s1dY$IE9a2I}vnT*m6AKI&z>;ruBcB(X4y;0)sMZ;k^Mx=yXcTodeO-BZ|t74&ixeO^sr#_4+rj
zB+NiF|2pABvl#%bu>BC|6Od+v%nOA(H#qWykCeN*QK_DAx}7c!ZT}jYINf%`KJ#fe
zI@2YedwvroDHaC1IJyPfGw4)34L7G{LCL};+uSTz$$#r!kT%nn>(ld3tOztMOYH~m
zru%6RdMq<(MV1X0s9y=}Z+%QWDAQ_;d}H?L_&Q)YsLW~CK-SW4!_`Nzho|7%OJEd|
z)O&k)^rN#3&Mviu;@Ukr=k?bCYTITFJy4x%YgMQ7MZ;o!e9*2%0MTmTY1ZG!yibvB
zIiB3WGdQNtU((64QR{4TG#7r}DiUu`Kmf2SxZ#nC;XVHOI(14z&)3%V8#H;;wi3rB
zl9ufAPClMx>_@rnD!r?7F=eR5woMhQw+b-|zas)#S@FmOyB1pftm!Kb+~!1BzGA#M
z?X%_h^E{s*IF%aR`3oRK0K*}KOi-5B@&h?p(--72$O
z?mRnl)J(i3Y1Z^~d~B73Mb^+#J5l3;0L1n)X5OZz)XVvkUQ6EEM$r-3-(8fQZzkaC
z+mt=M_B_)w8f)bpIcnPXkC=iDmrdKvAd3#-y-`tIhaY<}*B
| 
|
-| MiniMind2 | 
| 
|
-
-### Training Completed - Model Collection
-
-> Considering that many people have reported slow speeds with Baidu Cloud, all MiniMind2 models and beyond will be
-> hosted on ModelScope/HuggingFace.
-
----
-
-#### ① Native PyTorch Models
-
-MiniMind2 model
-weights ([ModelScope](https://www.modelscope.cn/models/gongjy/MiniMind2-PyTorch) | [HuggingFace](https://huggingface.co/jingyaogong/MiniMind2-Pytorch))
-
-MiniMind-V1 model weights ([Baidu Pan](https://pan.baidu.com/s/1KUfSzEkSXYbCCBj0Pw-9fA?pwd=6666))
-
-
-
-
-
-
-
-
-
-
-## 😊Acknowledgments
-
-@ipfgao:
-🔗Training steps record
-
-@chuanzhubin:
-🔗Line-by-line code comments
-
-@WangRongsheng:
-🔗Large dataset preprocessing
-
-@pengqianhan:
-🔗A brief tutorial
-
-@RyanSunn:
-🔗Inference process learning record
-
-@Nijikadesu:
-🔗Decompose project code in an interactive notebook format
-
-
- 
-