New Post: Linq-To-Sql sourced IQueryable IMediaFile query not executing efficiently via DataFacadeQueryable.

Hi All;

Within the composite code there are many bits querying the IMediaFile and IMediaFileFolder data items. For instance, there is the following query within the MediaUrls class:

        private static IMediaFile GetFileById(string storeId, Guid fileId)
        {
            using (new DataScope(DataScopeIdentifier.Public))
            {
                var query = DataFacade.GetData<IMediaFile>();

                if (query.IsEnumerableQuery())
                {
                    return (query as IEnumerable<IMediaFile>)
                        .FirstOrDefault(f => f.StoreId == storeId && f.Id == fileId);
                }

                return query
                    .FirstOrDefault(f => f.StoreId == storeId && f.Id == fileId);
            }
        }

I have written a custom media provider, which returns a Linq-To-Sql IQueryable collection.

If I manually create an instance of this media provider (for testing) and execute a typical expression against it (like the one in the example above) the correct sql statements are generated.

For e.g.:

        var myMediaProvider = new MyMediaProvider();

        var query = myMediaProvider.GetData<IMediaFile>();

        var file = query
                    .FirstOrDefault(f => f.StoreId == storeId && f.Id == fileId);

Produces the following sql (via sql tracer):

exec sp_executesql N'SELECT TOP (1) [t0].[Id], [t0].[KeyPath], [t0].[CompositePath], [t0].[StoreId], [t0].[FolderPath], [t0].[FileName], [t0].[Title], [t0].[Culture], [t0].[MimeType], [t0].[Description], [t0].[Length], [t0].[CreationTime], [t0].[LastWriteTime], [t0].[LocalAssetId], [t0].[Profile]
FROM [dbo].[MyMediaStore_MyMediaFile] AS [t0]
WHERE ([t0].[StoreId] = @p0) AND ([t0].[Id] = @p1)',N'@p0 nvarchar(4000),@p1 uniqueidentifier',@p0=N'MyMediaStore',@p1='3C48DE78-8F22-431E-8458-076A0D861F25'

Unfortunately, when executing via the composite system things don't work out as nicely for me. When executed through the standard composite workflow my Linq-To-Sql query gets evaluated at it's base (meaning all records get pulled from the db, which in my case is 1000s), before the the filter expression gets evaluated.

I tried to debug through the Composite code to see how and where this happens.

The query seems to be fetched via the DataFacadeImpl class, which fetches all the registered providers for a target IData type (in this case IMediaFile), and then wraps them all into a single DataFacadeQueryable<T> instance. Extract of this code is here:

        public IQueryable<T> GetData<T>(bool useCaching, IEnumerable<string> providerNames)
            where T : class, IData
        {
            IQueryable<T> resultQueryable;

            if (DataProviderRegistry.AllInterfaces.Contains(typeof(T)))
            {
                if (useCaching && DataCachingFacade.IsDataAccessCacheEnabled(typeof(T)))
                {
                    resultQueryable = DataCachingFacade.GetDataFromCache<T>();
                }
                else
                {
                    if (providerNames == null)
                    {
                        providerNames = DataProviderRegistry.GetDataProviderNamesByInterfaceType(typeof(T));
                    }

                    List<IQueryable<T>> queryables = new List<IQueryable<T>>();
                    foreach (string providerName in providerNames)
                    {
                        IQueryable<T> queryable = DataProviderPluginFacade.GetData<T>(providerName);

                        queryables.Add(queryable);
                    }

                    DataFacadeQueryable<T> multibleSourceQueryable = new DataFacadeQueryable<T>(queryables);

                    resultQueryable = multibleSourceQueryable;
                }
            }

Oki, then when the query gets evaluated the following code is hit in the DataFacadeQueryable code:

        public S Execute<S>(Expression expression)
        {
            bool pullIntoMemory = ShouldBePulledIntoMemory(expression);

            DataFacadeQueryableExpressionVisitor handleInProviderVisitor =
                new DataFacadeQueryableExpressionVisitor(pullIntoMemory);

            Expression newExpression = handleInProviderVisitor.Visit(expression);

            IQueryable source = handleInProviderVisitor.Queryable;

            return (S)source.Provider.Execute(newExpression);
        }

The code is quite complex at this point - I am still trying to step through it and understand it's flow. I looked briefly at the DataFacadeQueryableExpressionVisitor and see that it has some specific code for handling different declaring type, one case being a standard DataConnection. An extract of the method from the visitor class is below:

        protected override Expression VisitMethodCall(MethodCallExpression m)
        {
            if (m.Method.DeclaringType == typeof (DataFacade))
            {
                if ((m.Method.IsGenericMethod) &&
                     (m.Method.GetGenericMethodDefinition() == _dataFacadeGetDataMethodInfo))
                {
                    object result = m.Method.Invoke(null, null);

                    return Expression.Constant(result is IDataFacadeQueryable ? HandleMultipleSourceQueryable(result) : result);
                }

                // Handling some of the overloads of GetData()
                if (m.Method.Name == _dataFacadeGetDataMethodInfo.Name && m.Arguments.All(arg => (arg as ConstantExpression) != null))
                {
                    object[] parameters = m.Arguments.Select(arg => (arg as ConstantExpression).Value).ToArray();

                    object result = m.Method.Invoke(null, parameters);

                    return Expression.Constant(result is IDataFacadeQueryable ? HandleMultipleSourceQueryable(result) : result);
                }
            
                throw new NotImplementedException("Supporing for DataFacade method '{0}' or one of it's overloads not yet implemented".FormatWith(m.Method.Name));
            }

            if (m.Method.DeclaringType == typeof (DataConnection))
            {
                if (m.Method.IsGenericMethod 
                    && m.Method.GetGenericMethodDefinition() == _dataConnectionGetDataMethodInfo)
                {
                    var dataConnection = EvaluateExpression<DataConnection>(m.Object);
                    object result = m.Method.Invoke(dataConnection, null);

                    return Expression.Constant(result is IDataFacadeQueryable ? HandleMultipleSourceQueryable(result) : result);
                }

                throw new NotImplementedException("Supporing for DataConnection method '{0}' or one of it's overloads not yet implemented".FormatWith(m.Method.Name));
            }

            // Replacing Guid.NewGuid() call with "newid()" sql statement
            if (m.Method.IsStatic && m.Method.DeclaringType == typeof(Guid) && m.Method.Name == "NewGuid"
                && _queryable != null)
            {
                var dataContext = GetContext(_queryable) as DataContextBase;
                if(dataContext != null)
                {
                    return Expression.Call(Expression.Constant(dataContext), DataContextBase.GetNewIdMethodInfo());
                }
            }

            return base.VisitMethodCall(m);
        }

Am I correct in assuming that for my Linq-to-Sql queries to execute efficiently I would have to edit the code at this point and place another condition in here to handle Linq-to-Sql source queries efficiently?

Thanks for any advice on this.