Nest AI Research - LLM & Tasks

Posted by Riino on
Nest AI Research - LLM & Tasks
Nest AI Research - LLM & Tasks
Nest AI Research - LLM & Tasks
Nest AI Research - LLM & Tasks

Nest AI Research - LLM & Tasks

 
Selected Key Researches based on LLM
 
SurveyTools:PLMsFactualityPapers - LLM EnhancementBackbones

Survey

A Survey of Large Language Models
Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and...
A Survey of Large Language Models
https://arxiv.org/abs/2303.18223
A Survey of Large Language Models
Survey on Factuality in Large Language Models: Knowledge,...
This survey addresses the crucial issue of factuality in Large Language Models (LLMs). As LLMs find applications across diverse domains, the reliability and accuracy of their outputs become vital....
Survey on Factuality in Large Language Models: Knowledge,...
https://arxiv.org/abs/2310.07521
Survey on Factuality in Large Language Models: Knowledge,...
notion image
A Survey of Large Language Models
Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and...
A Survey of Large Language Models
https://arxiv.org/abs/2303.18223
A Survey of Large Language Models
LLM family survey (LLM Summary)
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
This paper presents a comprehensive and practical guide for practitioners and end-users working with Large Language Models (LLMs) in their downstream natural language processing (NLP) tasks. We...
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
https://arxiv.org/abs/2304.13712
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
notion image

Tools:

PLMs

Colossal (based on LLAMA)
Falcon
tiiuae/falcon-7b · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
tiiuae/falcon-7b · Hugging Face
https://huggingface.co/tiiuae/falcon-7b
tiiuae/falcon-7b · Hugging Face
LaMini-Flan-T5
MBZUAI/LaMini-Flan-T5-783M · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
MBZUAI/LaMini-Flan-T5-783M · Hugging Face
https://huggingface.co/MBZUAI/LaMini-Flan-T5-783M
MBZUAI/LaMini-Flan-T5-783M · Hugging Face
Stable
Vicuna : 99% GPT-3.5
lmsys/vicuna-13b-delta-v0 · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
lmsys/vicuna-13b-delta-v0 · Hugging Face
https://huggingface.co/lmsys/vicuna-13b-delta-v0
lmsys/vicuna-13b-delta-v0 · Hugging Face
Dolly V2 (Pythia) : Business Lisence LLaMA
databricks/dolly-v2-12b · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
databricks/dolly-v2-12b · Hugging Face
https://huggingface.co/databricks/dolly-v2-12b
databricks/dolly-v2-12b · Hugging Face

Factuality

CONNER COmpreheNsive kNowledge Evaluation fRamework
Beyond Factuality: A Comprehensive Evaluation of Large Language...
Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks when being prompted to generate world knowledge. However, community concerns...
Beyond Factuality: A Comprehensive Evaluation of Large Language...
https://arxiv.org/abs/2310.07289
Beyond Factuality: A Comprehensive Evaluation of Large Language...
Are You Sure That This Happened? Assessing the Factuality Degree of Events in Text
Abstract. Identifying the veracity, or factuality, of event mentions in text is fundamental for reasoning about eventualities in discourse. Inferences derived from events judged as not having happened, or as being only possible, are different from those derived from events evaluated as factual. Event factuality involves two separate levels of information. On the one hand, it deals with polarity, which distinguishes between positive and negative instantiations of events. On the other, it has to do with degrees of certainty (e.g., possible, probable), an information level generally subsumed under the category of epistemic modality. This article aims at contributing to a better understanding of how event factuality is articulated in natural language. For that purpose, we put forward a linguistic-oriented computational model which has at its core an algorithm articulating the effect of factuality relations across levels of syntactic embedding. As a proof of concept, this model has been implemented in De Facto, a factuality profiler for eventualities mentioned in text, and tested against a corpus built specifically for the task, yielding an F1 of 0.70 (macro-averaging) and 0.80 (micro-averaging). These two measures mutually compensate for an over-emphasis present in the other (either on the lesser or greater populated categories), and can therefore be interpreted as the lower and upper bounds of the De Facto's performance.
Are You Sure That This Happened? Assessing the Factuality Degree of Events in Text
https://direct.mit.edu/coli/article/38/2/261/2145/Are-You-Sure-That-This-Happened-Assessing-the
Are You Sure That This Happened? Assessing the Factuality Degree of Events in Text
Fact-checking LLM-generated content
In this article we’ll dive deeper into everything you need to know about fact-checking LLM-generated content, including why it’s important to fact-check content, how to verify accuracy, what to pay attention to, and more.
Fact-checking LLM-generated content
https://toloka.ai/blog/fact-checking-llm-generated-content/
Fact-checking LLM-generated content
Entity-Based Knowledge Conflicts in Question Answering
Knowledge-dependent tasks typically use two sources of knowledge: parametric, learned at training time, and contextual, given as a passage at inference time. To understand how models use these...
Entity-Based Knowledge Conflicts in Question Answering
https://arxiv.org/abs/2109.05052
Entity-Based Knowledge Conflicts in Question Answering
On the Risk of Misinformation Pollution with Large Language Models
In this paper, we comprehensively investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation and its subsequent impact on...
On the Risk of Misinformation Pollution with Large Language Models
https://arxiv.org/abs/2305.13661
On the Risk of Misinformation Pollution with Large Language Models
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their...
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
https://arxiv.org/abs/2005.11401
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning. In particular,...
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
https://arxiv.org/abs/2201.11903
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation
Retrieval-augmented generation models offer many benefits over standalone language models: besides a textual answer to a given query they provide provenance items retrieved from an updateable...
FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation
https://arxiv.org/abs/2209.14290
FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation
Generative Language Models and Automated Influence Operations:...
Generative language models have improved drastically, and can now produce realistic text outputs that are difficult to distinguish from human-written content. For malicious actors, these language...
Generative Language Models and Automated Influence Operations:...
https://arxiv.org/abs/2301.04246
Generative Language Models and Automated Influence Operations:...

Papers - LLM Enhancement

[Text-to-SQL]
Know What I don't Know: Handling Ambiguous and Unanswerable...
The task of text-to-SQL aims to convert a natural language question into its corresponding SQL query within the context of relational tables. Existing text-to-SQL parsers generate a "plausible"...
Know What I don't Know: Handling Ambiguous and Unanswerable...
https://arxiv.org/abs/2212.08902
Know What I don't Know: Handling Ambiguous and Unanswerable...
GNN to Transformer
Transformers are Graph Neural Networks | NTU Graph Deep Learning Lab
Engineer friends often ask me: Graph Deep Learning sounds great, but are there any big commercial success stories? Is it being deployed in practical applications? Besides the obvious ones–recommendation systems at Pinterest, Alibaba and Twitter–a slightly nuanced success story is the Transformer architecture, which has taken the NLP industry by storm. Through this post, I want to establish links between Graph Neural Networks (GNNs) and Transformers. I’ll talk about the intuitions behind model architectures in the NLP and GNN communities, make connections using equations and figures, and discuss how we could work together to drive progress.
Transformers are Graph Neural Networks | NTU Graph Deep Learning Lab
https://graphdeeplearning.github.io/post/transformers-are-gnns/
Transformers are Graph Neural Networks | NTU Graph Deep Learning Lab
LLM As DBA
Database administrators (DBAs) play a crucial role in managing, maintaining and optimizing a database system to ensure data availability, performance, and reliability. However, it is hard and...
LLM As DBA
https://arxiv.org/abs/2308.05481
LLM As DBA
[Multimodal] Single Diffuser for Video/Voice/Text/Image
Any-to-Any Generation via Composable Diffusion
We present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such as language, image, video, or audio, from any combination of input...
Any-to-Any Generation via Composable Diffusion
https://arxiv.org/abs/2305.11846
Any-to-Any Generation via Composable Diffusion
notion image
[Model Structurel] RNN + Transfermer
RWKV: Reinventing RNNs for the Transformer Era
Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence length. In contrast,...
RWKV: Reinventing RNNs for the Transformer Era
https://arxiv.org/abs/2305.13048
RWKV: Reinventing RNNs for the Transformer Era
notion image
[Multimodal] Output Keyboard/Mouse Opeartion Command (Recursive Criticism and Improvement (RCI))
Language Models can Solve Computer Tasks
Agents capable of carrying out general tasks on a computer can improve efficiency and productivity by automating repetitive tasks and assisting in complex problem-solving. Ideally, such agents...
Language Models can Solve Computer Tasks
https://arxiv.org/abs/2303.17491
Language Models can Solve Computer Tasks
arxiv.org
https://arxiv.org/pdf/2303.17491.pdf
[Multimodal] HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face(JARVIS)
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace
Solving complicated AI tasks with different domains and modalities is a key step toward advanced artificial intelligence. While there are abundant AI models available for different domains and...
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace
https://arxiv.org/abs/2303.17580
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace
arxiv.org
https://arxiv.org/pdf/2303.17580.pdf
[Model Structurel] ‘My story has several episodes, kindly watch the ‘previously on’ (Amplify input length)
Scaling Transformer to 1M tokens and beyond with RMT
This technical report presents the application of a recurrent memory to extend the context length of BERT, one of the most effective Transformer-based models in natural language processing. By...
Scaling Transformer to 1M tokens and beyond with RMT
https://arxiv.org/abs/2304.11062
Scaling Transformer to 1M tokens and beyond with RMT
arxiv.org
https://arxiv.org/pdf/2304.11062.pdf
‘GPT-4 will judge you, rather than me.’ (LLM benchmark strategy)
Instruction Tuning with GPT-4
Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilities on new tasks,...
Instruction Tuning with GPT-4
https://arxiv.org/abs/2304.03277
Instruction Tuning with GPT-4
notion image
‘You guys talk with each other’ (GAN-like LLM tuning)
DERA: Enhancing Large Language Model Completions with...
Large language models (LLMs) have emerged as valuable tools for many natural language understanding tasks. In safety-critical applications such as healthcare, the utility of these models is...
DERA: Enhancing Large Language Model Completions with...
https://arxiv.org/abs/2303.17071
DERA: Enhancing Large Language Model Completions with...
‘Look at what you just said.’ (Key concept of AutoGPT family’)
Reflexion: an autonomous agent with dynamic memory and self-reflection
Recent advancements in decision-making large language model (LLM) agents have demonstrated impressive performance across various benchmarks. However, these state-of-the-art approaches typically...
Reflexion: an autonomous agent with dynamic memory and self-reflection
https://arxiv.org/abs/2303.11366
Reflexion: an autonomous agent with dynamic memory and self-reflection
‘You can join the exam many times’ (Self-consistency optimizing)
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy,...
Self-Consistency Improves Chain of Thought Reasoning in Language Models
https://arxiv.org/abs/2203.11171
Self-Consistency Improves Chain of Thought Reasoning in Language Models

Backbones

[Interpretability]Elementary Proven of Relationships between Neurons in LLM and Knowlege Relationships (Selected Relationships in Knowlege Graph) by supressing or amplifying ‘knowledge neurons’ while observing Knowledge Attribution
Knowledge Neurons in Pretrained Transformers
Large-scale pretrained language models are surprisingly good at recalling factual knowledge presented in the training corpus. In this paper, we present preliminary studies on how factual knowledge...
Knowledge Neurons in Pretrained Transformers
https://arxiv.org/abs/2104.08696
Knowledge Neurons in Pretrained Transformers
notion image
A new Gradient descent method
Sophia: A Scalable Stochastic Second-order Optimizer for Language...
Given the massive cost of language model pre-training, a non-trivial improvement of the optimization algorithm would lead to a material reduction on the time and cost of training. Adam and its...
Sophia: A Scalable Stochastic Second-order Optimizer for Language...
https://arxiv.org/abs/2305.14342
Sophia: A Scalable Stochastic Second-order Optimizer for Language...
notion image
[Model Structurel]Soft Tuning
The Power of Scale for Parameter-Efficient Prompt Tuning
In this work, we explore "prompt tuning", a simple yet effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. Unlike the discrete...
The Power of Scale for Parameter-Efficient Prompt Tuning
https://arxiv.org/abs/2104.08691
The Power of Scale for Parameter-Efficient Prompt Tuning
notion image
[Model Structurel]Attention
Attention Is All You Need
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder...
Attention Is All You Need
https://arxiv.org/abs/1706.03762
Attention Is All You Need
notion image