Sync to upstream/release/665 (#1732)

Hello all! Another week, another Luau release!

# Change to `lua_setuserdatametatable`

This release fixes #1710: `lua_setuserdatametatable` is being changed so
that it _only_ operates on the top of the stack: the `idx` parameter is
being removed. Prior to this, `lua_setuserdatametable` would set the
metatable of the value in the stack at `idx`, but _always_ pop the top
of the stack. The old behavior is available in this release as
`lua_setuserdatametatable_DEPRECATED`.

# General

This release exposes a generalized implementation of require-by-string's
autocomplete logic. `FileResolver` can now be optionally constructed
with a `RequireSuggester`, which provides an interface for converting a
given module to a `RequireNode`. Consumers of this new API implement a
`RequireNode` to define how modules are represented in their embedded
context, and the new API manages the logic specific to
require-by-string, including providing suggestions for require aliases.
This enhancement moves toward integrating require-by-string's semantics
into the language itself, rather than merely providing a specification
for community members to implement themselves.

# New Type Solver
* Fixed a source of potential `Luau::follow detected a Type cycle`
internal compiler exceptions when assigning a global to itself.
* Fixed an issue whereby `*no-refine*` (a type which should not be
visible at the end of type checking) was not being properly elided,
causing inference of class-like tables to become unreadable / induce
crashes in autocomplete.
* Fixed a case of incomplete constraint solving when performing basic
math in a loop

# Fragment Autocomplete
* Fixed several crashes related to not properly filling in scope
information for the fragments
* Fixed a source of memory corruption by isolating the return type of a
fragment when it is type checked.
* Improved performance by opting not to clone persistent types for the
fragment (e.g.: built in types)
 
# Internal Contributors
Co-authored-by: Andy Friesen <[email protected]>
Co-authored-by: Hunter Goldstein <[email protected]>
Co-authored-by: Talha Pathan <[email protected]>
Co-authored-by: Varun Saini <[email protected]>
Co-authored-by: Vighnesh Vijay <[email protected]>
Co-authored-by: Vyacheslav Egorov <[email protected]>

---------

Co-authored-by: Varun Saini <[email protected]>
Co-authored-by: Alexander Youngblood <[email protected]>
Co-authored-by: Menarul Alam <[email protected]>
Co-authored-by: Aviral Goel <[email protected]>
Co-authored-by: Vighnesh <[email protected]>
Co-authored-by: Vyacheslav Egorov <[email protected]>
Co-authored-by: Ariel Weiss <[email protected]>

Author: 8 people

Analysis/include/Luau/FileResolver.h

@@ -1,8 +1,9 @@
 // This file is part of the Luau programming language and is licensed under MIT License; see LICENSE.txt for details
 #pragma once
 
-#include <string>
+#include <memory>
 #include <optional>
+#include <string>
 #include <vector>
 
 namespace Luau
@@ -32,15 +33,71 @@ struct ModuleInfo
     bool optional = false;
 };
 
+struct RequireAlias
+{
+    std::string alias; // Unprefixed alias name (no leading `@`).
+    std::vector<std::string> tags = {};
+};
+
+struct RequireNode
+{
+    virtual ~RequireNode() {}
+
+    // Get the path component representing this node.
+    virtual std::string getPathComponent() const = 0;
+
+    // Get the displayed user-facing label for this node, defaults to getPathComponent()
+    virtual std::string getLabel() const
+    {
+        return getPathComponent();
+    }
+
+    // Get tags to attach to this node's RequireSuggestion (defaults to none).
+    virtual std::vector<std::string> getTags() const
+    {
+        return {};
+    }
+
+    // TODO: resolvePathToNode() can ultimately be replaced with a call into
+    // require-by-string's path resolution algorithm. This will first require
+    // generalizing that algorithm to work with a virtual file system.
+    virtual std::unique_ptr<RequireNode> resolvePathToNode(const std::string& path) const = 0;
+
+    // Get children of this node, if any (if this node represents a directory).
+    virtual std::vector<std::unique_ptr<RequireNode>> getChildren() const = 0;
+
+    // A list of the aliases available to this node.
+    virtual std::vector<RequireAlias> getAvailableAliases() const = 0;
+};
+
 struct RequireSuggestion
 {
     std::string label;
     std::string fullPath;
+    std::vector<std::string> tags;
 };
 using RequireSuggestions = std::vector<RequireSuggestion>;
 
+struct RequireSuggester
+{
+    virtual ~RequireSuggester() {}
+    std::optional<RequireSuggestions> getRequireSuggestions(const ModuleName& requirer, const std::optional<std::string>& pathString) const;
+
+protected:
+    virtual std::unique_ptr<RequireNode> getNode(const ModuleName& name) const = 0;
+
+private:
+    std::optional<RequireSuggestions> getRequireSuggestionsImpl(const ModuleName& requirer, const std::optional<std::string>& path) const;
+};
+
 struct FileResolver
 {
+    FileResolver() = default;
+    FileResolver(std::shared_ptr<RequireSuggester> requireSuggester)
+        : requireSuggester(std::move(requireSuggester))
+    {
+    }
+
     virtual ~FileResolver() {}
 
     virtual std::optional<SourceCode> readSource(const ModuleName& name) = 0;
@@ -60,10 +117,10 @@ struct FileResolver
         return std::nullopt;
     }
 
-    virtual std::optional<RequireSuggestions> getRequireSuggestions(const ModuleName& requirer, const std::optional<std::string>& pathString) const
-    {
-        return std::nullopt;
-    }
+    // Make non-virtual when removing FFlagLuauImproveRequireByStringAutocomplete.
+    virtual std::optional<RequireSuggestions> getRequireSuggestions(const ModuleName& requirer, const std::optional<std::string>& pathString) const;
+
+    std::shared_ptr<RequireSuggester> requireSuggester;
 };
 
 struct NullFileResolver : FileResolver

Analysis/include/Luau/FragmentAutocomplete.h

@@ -15,6 +15,28 @@ namespace Luau
 {
 struct FrontendOptions;
 
+enum class FragmentAutocompleteWaypoint
+{
+    ParseFragmentEnd,
+    CloneModuleStart,
+    CloneModuleEnd,
+    DfgBuildEnd,
+    CloneAndSquashScopeStart,
+    CloneAndSquashScopeEnd,
+    ConstraintSolverStart,
+    ConstraintSolverEnd,
+    TypecheckFragmentEnd,
+    AutocompleteEnd,
+    COUNT,
+};
+
+class IFragmentAutocompleteReporter
+{
+public:
+    virtual void reportWaypoint(FragmentAutocompleteWaypoint) = 0;
+    virtual void reportFragmentString(std::string_view) = 0;
+};
+
 enum class FragmentTypeCheckStatus
 {
     SkipAutocomplete,
@@ -70,7 +92,8 @@ std::pair<FragmentTypeCheckStatus, FragmentTypeCheckResult> typecheckFragment(
     const Position& cursorPos,
     std::optional<FrontendOptions> opts,
     std::string_view src,
-    std::optional<Position> fragmentEndPosition
+    std::optional<Position> fragmentEndPosition,
+    IFragmentAutocompleteReporter* reporter = nullptr
 );
 
 FragmentAutocompleteResult fragmentAutocomplete(
@@ -80,7 +103,8 @@ FragmentAutocompleteResult fragmentAutocomplete(
     Position cursorPosition,
     std::optional<FrontendOptions> opts,
     StringCompletionCallback callback,
-    std::optional<Position> fragmentEndPosition = std::nullopt
+    std::optional<Position> fragmentEndPosition = std::nullopt,
+    IFragmentAutocompleteReporter* reporter = nullptr
 );
 
 enum class FragmentAutocompleteStatus
@@ -102,6 +126,7 @@ struct FragmentContext
     const ParseResult& freshParse;
     std::optional<FrontendOptions> opts;
     std::optional<Position> DEPRECATED_fragmentEndPosition;
+    IFragmentAutocompleteReporter* reporter = nullptr;
 };
 
 /**

Analysis/include/Luau/Quantify.h

@@ -31,13 +31,4 @@ struct OrderedMap
     }
 };
 
-struct QuantifierResult
-{
-    TypeId result;
-    OrderedMap<TypeId, TypeId> insertedGenerics;
-    OrderedMap<TypePackId, TypePackId> insertedGenericPacks;
-};
-
-std::optional<QuantifierResult> quantify(TypeArena* arena, TypeId ty, Scope* scope);
-
 } // namespace Luau

Analysis/src/AutocompleteCore.cpp

@@ -25,6 +25,8 @@ LUAU_FASTINT(LuauTypeInferIterationLimit)
 LUAU_FASTINT(LuauTypeInferRecursionLimit)
 LUAU_FASTFLAGVARIABLE(DebugLuauMagicVariableNames)
 
+LUAU_FASTFLAG(LuauExposeRequireByStringAutocomplete)
+
 LUAU_FASTFLAGVARIABLE(LuauAutocompleteRefactorsForIncrementalAutocomplete)
 
 LUAU_FASTFLAGVARIABLE(LuauAutocompleteUsesModuleForTypeCompatibility)
@@ -1519,10 +1521,14 @@ static std::optional<AutocompleteEntryMap> convertRequireSuggestionsToAutocomple
         return std::nullopt;
 
     AutocompleteEntryMap result;
-    for (const RequireSuggestion& suggestion : *suggestions)
+    for (RequireSuggestion& suggestion : *suggestions)
     {
         AutocompleteEntry entry = {AutocompleteEntryKind::RequirePath};
         entry.insertText = std::move(suggestion.fullPath);
+        if (FFlag::LuauExposeRequireByStringAutocomplete)
+        {
+            entry.tags = std::move(suggestion.tags);
+        }
         result[std::move(suggestion.label)] = std::move(entry);
     }
     return result;

Analysis/src/Clone.cpp

@@ -1,6 +1,7 @@
 // This file is part of the Luau programming language and is licensed under MIT License; see LICENSE.txt for details
 #include "Luau/Clone.h"
 
+#include "Luau/Common.h"
 #include "Luau/NotNull.h"
 #include "Luau/Type.h"
 #include "Luau/TypePack.h"
@@ -12,6 +13,8 @@ LUAU_FASTFLAG(LuauFreezeIgnorePersistent)
 
 // For each `Luau::clone` call, we will clone only up to N amount of types _and_ packs, as controlled by this limit.
 LUAU_FASTINTVARIABLE(LuauTypeCloneIterationLimit, 100'000)
+LUAU_FASTFLAGVARIABLE(LuauClonedTableAndFunctionTypesMustHaveScopes)
+LUAU_FASTFLAGVARIABLE(LuauDoNotClonePersistentBindings)
 
 namespace Luau
 {
@@ -477,7 +480,7 @@ class TypeCloner
 
 class FragmentAutocompleteTypeCloner final : public TypeCloner
 {
-    Scope* freeTypeReplacementScope = nullptr;
+    Scope* replacementForNullScope = nullptr;
 
 public:
     FragmentAutocompleteTypeCloner(
@@ -487,12 +490,12 @@ class FragmentAutocompleteTypeCloner final : public TypeCloner
         NotNull<SeenTypePacks> packs,
         TypeId forceTy,
         TypePackId forceTp,
-        Scope* freeTypeReplacementScope
+        Scope* replacementForNullScope
     )
         : TypeCloner(arena, builtinTypes, types, packs, forceTy, forceTp)
-        , freeTypeReplacementScope(freeTypeReplacementScope)
+        , replacementForNullScope(replacementForNullScope)
     {
-        LUAU_ASSERT(freeTypeReplacementScope);
+        LUAU_ASSERT(replacementForNullScope);
     }
 
     TypeId shallowClone(TypeId ty) override
@@ -512,12 +515,18 @@ class FragmentAutocompleteTypeCloner final : public TypeCloner
             generic->scope = nullptr;
         else if (auto free = getMutable<FreeType>(target))
         {
-            free->scope = freeTypeReplacementScope;
+            free->scope = replacementForNullScope;
+        }
+        else if (auto tt = getMutable<TableType>(target))
+        {
+            if (FFlag::LuauClonedTableAndFunctionTypesMustHaveScopes)
+                tt->scope = replacementForNullScope;
         }
         else if (auto fn = getMutable<FunctionType>(target))
-            fn->scope = nullptr;
-        else if (auto table = getMutable<TableType>(target))
-            table->scope = nullptr;
+        {
+            if (FFlag::LuauClonedTableAndFunctionTypesMustHaveScopes)
+                fn->scope = replacementForNullScope;
+        }
 
         (*types)[ty] = target;
         queue.emplace_back(target);
@@ -538,7 +547,7 @@ class FragmentAutocompleteTypeCloner final : public TypeCloner
         if (auto generic = getMutable<GenericTypePack>(target))
             generic->scope = nullptr;
         else if (auto free = getMutable<FreeTypePack>(target))
-            free->scope = freeTypeReplacementScope;
+            free->scope = replacementForNullScope;
 
         (*packs)[tp] = target;
         queue.emplace_back(target);
@@ -728,7 +737,7 @@ Binding cloneIncremental(const Binding& binding, TypeArena& dest, CloneState& cl
     b.deprecatedSuggestion = binding.deprecatedSuggestion;
     b.documentationSymbol = binding.documentationSymbol;
     b.location = binding.location;
-    b.typeId = cloner.clone(binding.typeId);
+    b.typeId = FFlag::LuauDoNotClonePersistentBindings && binding.typeId->persistent ? binding.typeId : cloner.clone(binding.typeId);
 
     return b;
 }

Analysis/src/ConstraintGenerator.cpp

@@ -38,10 +38,12 @@ LUAU_FASTFLAG(DebugLuauGreedyGeneralization)
 LUAU_FASTFLAGVARIABLE(LuauTrackInteriorFreeTypesOnScope)
 LUAU_FASTFLAGVARIABLE(LuauDeferBidirectionalInferenceForTableAssignment)
 LUAU_FASTFLAGVARIABLE(LuauUngeneralizedTypesForRecursiveFunctions)
+LUAU_FASTFLAGVARIABLE(LuauGlobalSelfAssignmentCycle)
 
 LUAU_FASTFLAG(LuauFreeTypesMustHaveBounds)
 LUAU_FASTFLAGVARIABLE(LuauInferLocalTypesInMultipleAssignments)
 LUAU_FASTFLAGVARIABLE(LuauDoNotLeakNilInRefinement)
+LUAU_FASTFLAGVARIABLE(LuauExtraFollows)
 
 namespace Luau
 {
@@ -2082,7 +2084,7 @@ InferencePack ConstraintGenerator::checkPack(const ScopePtr& scope, AstExprCall*
         {
             std::vector<TypeId> unpackedTypes;
             if (args.size() > 0)
-                target = args[0];
+                target = FFlag::LuauExtraFollows ? follow(args[0]) : args[0];
             else
             {
                 target = arena->addType(BlockedType{});
@@ -2891,6 +2893,13 @@ void ConstraintGenerator::visitLValue(const ScopePtr& scope, AstExprGlobal* glob
         DefId def = dfg->getDef(global);
         rootScope->lvalueTypes[def] = rhsType;
 
+        if (FFlag::LuauGlobalSelfAssignmentCycle)
+        {
+            // Ignore possible self-assignment, it doesn't create a new constraint
+            if (annotatedTy == follow(rhsType))
+                return;
+        }
+
         // Sketchy: We're specifically looking for BlockedTypes that were
         // initially created by ConstraintGenerator::prepopulateGlobalScope.
         if (auto bt = get<BlockedType>(follow(*annotatedTy)); bt && !bt->getOwner())

Analysis/src/ConstraintSolver.cpp

@@ -37,6 +37,7 @@ LUAU_FASTFLAG(LuauTrackInteriorFreeTypesOnScope)
 LUAU_FASTFLAGVARIABLE(LuauTrackInteriorFreeTablesOnScope)
 LUAU_FASTFLAGVARIABLE(LuauPrecalculateMutatedFreeTypes2)
 LUAU_FASTFLAGVARIABLE(DebugLuauGreedyGeneralization)
+LUAU_FASTFLAG(LuauSearchForRefineableType)
 
 namespace Luau
 {
@@ -907,26 +908,16 @@ bool ConstraintSolver::tryDispatch(const GeneralizationConstraint& c, NotNull<co
     else if (get<PendingExpansionType>(generalizedType))
         return block(generalizedType, constraint);
 
-    std::optional<QuantifierResult> generalized;
-
     std::optional<TypeId> generalizedTy = generalize(NotNull{arena}, builtinTypes, constraint->scope, generalizedTypes, c.sourceType);
-    if (generalizedTy)
-        generalized = QuantifierResult{*generalizedTy}; // FIXME insertedGenerics and insertedGenericPacks
-    else
+    if (!generalizedTy)
         reportError(CodeTooComplex{}, constraint->location);
 
-    if (generalized)
+    if (generalizedTy)
     {
         if (get<BlockedType>(generalizedType))
-            bind(constraint, generalizedType, generalized->result);
+            bind(constraint, generalizedType, *generalizedTy);
         else
-            unify(constraint, generalizedType, generalized->result);
-
-        for (auto [free, gen] : generalized->insertedGenerics.pairings)
-            unify(constraint, free, gen);
-
-        for (auto [free, gen] : generalized->insertedGenericPacks.pairings)
-            unify(constraint, free, gen);
+            unify(constraint, generalizedType, *generalizedTy);
     }
     else
     {
@@ -1356,15 +1347,29 @@ void ConstraintSolver::fillInDiscriminantTypes(NotNull<const Constraint> constra
         if (!ty)
             continue;
 
-        // If the discriminant type has been transmuted, we need to unblock them.
-        if (!isBlocked(*ty))
+        if (FFlag::LuauSearchForRefineableType)
         {
+            if (isBlocked(*ty))
+                // We bind any unused discriminants to the `*no-refine*` type indicating that it can be safely ignored.
+                emplaceType<BoundType>(asMutable(follow(*ty)), builtinTypes->noRefineType);
+            
+            // We also need to unconditionally unblock these types, otherwise
+            // you end up with funky looking "Blocked on *no-refine*."
             unblock(*ty, constraint->location);
-            continue;
         }
+        else
+        {
 
-        // We bind any unused discriminants to the `*no-refine*` type indicating that it can be safely ignored.
-        emplaceType<BoundType>(asMutable(follow(*ty)), builtinTypes->noRefineType);
+            // If the discriminant type has been transmuted, we need to unblock them.
+            if (!isBlocked(*ty))
+            {
+                unblock(*ty, constraint->location);
+                continue;
+            }
+
+            // We bind any unused discriminants to the `*no-refine*` type indicating that it can be safely ignored.
+            emplaceType<BoundType>(asMutable(follow(*ty)), builtinTypes->noRefineType);
+        }
     }
 }